Medical scan labeling quality assurance system

ABSTRACT

A medical scan labeling quality assurance system is operable to transmit a selected set of medical scans to a set of client devices associated with an expert user and a selected set of users. The client devices display medical scans are displayed to the expert user and the set of users, and a set of labeling data generated via user input to each client device is received from each client device. A set of performance score data is generated based on comparing each set of labeling data to a set of golden labeling data that was received from the client device of the expert user. The set of performance score data is used to update user profiles of the set of users, and is transmitted to the set of client devices for display to the set of users.

CROSS REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 119(e) to U.S. Provisional Application No. 62/770,334,entitled “LESION TRACKING SYSTEM”, filed Nov. 21, 2018, which is herebyincorporated herein by reference in its entirety and made part of thepresent U.S. Utility patent application for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND Technical Field

This invention relates generally to medical imaging devices andknowledge-based systems used in conjunction with client/server networkarchitectures.

DESCRIPTION OF RELATED ART Brief Description of the Several Views of theDrawing(s)

FIG. 1 is a schematic block diagram of an embodiment of a medical scanprocessing system;

FIG. 2A is a schematic block diagram of a client device in accordancewith various embodiments;

FIG. 2B is a schematic block diagram of one or more subsystems inaccordance with various embodiments;

FIG. 3 is a schematic block diagram of a database storage system inaccordance with various embodiments;

FIG. 4A is schematic block diagram of a medical scan entry in accordancewith various embodiments;

FIG. 4B is a schematic block diagram of abnormality data in accordancewith various embodiments;

FIG. 5A is a schematic block diagram of a user profile entry inaccordance with various embodiments;

FIG. 5B is a schematic block diagram of a medical scan analysis functionentry in accordance with various embodiments;

FIGS. 6A-6B are schematic block diagram of a medical scan diagnosingsystem in accordance with various embodiments;

FIG. 7A is a flowchart representation of an inference step in accordancewith various embodiments;

FIG. 7B is a flowchart representation of a detection step in accordancewith various embodiments;

FIGS. 8A-8F are schematic block diagrams of a medical picture archiveintegration system in accordance with various embodiments;

FIG. 9 is a flowchart representation of a method for execution by amedical picture archive integration system in accordance with variousembodiments;

FIG. 10A is a schematic block diagram of a de-identification system inaccordance with various embodiments;

FIG. 10B is an illustration of an example of anonymizing patientidentifiers in image data of a medical scan in accordance with variousembodiments;

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system in accordance with various embodiments;

FIGS. 12A-12B are schematic block diagrams of a medical scanhierarchical labeling system in accordance with various embodiments;

FIG. 12C is an illustration of an example of a diagnosis prompt decisiontree in accordance with various embodiments;

FIG. 12D is an illustration of an example of a characterization promptdecision tree in accordance with various embodiments;

FIG. 12E is an illustration of an example of a localization promptdecision tree in accordance with various embodiments;

FIGS. 12F-12G are schematic block diagrams of a medical scanhierarchical labeling system in accordance with various embodiments;

FIGS. 13A-13E are schematic block diagrams of a medical scan labelingquality assurance system in accordance with various embodiments;

FIGS. 14A-14B are schematic block diagrams of a medical scan annotatingsystem in accordance with various embodiments;

FIGS. 14C-14V are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIG. 15A presents a flowchart illustrating a method for execution by amedical scan hierarchical labeling system in accordance with variousembodiments;

FIG. 15B presents a flowchart illustrating a method for execution by aclient device in accordance with various embodiments; and

FIG. 16 presents a flowchart illustrating a method for execution by amedical scan labeling quality assurance system in accordance withvarious embodiments;

DETAILED DESCRIPTION

The present U.S. Utility patent application is related to U.S. Utilityapplication Ser. No. 15/627,644, entitled “MEDICAL SCAN ASSISTED REVIEWSYSTEM”, filed 20 Jun. 2017, which claims priority pursuant to 35 U.S.C.§ 119(e) to U.S. Provisional Application No. 62/511,150, entitled“MEDICAL SCAN ASSISTED REVIEW SYSTEM AND METHODS”, filed 25 May 2017,both of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility patent applicationfor all purposes.

FIG. 1 presents a medical scan processing system 100, which can includeone or more medical scan subsystems 101 that communicate bidirectionallywith one or more client devices 120 via a wired and/or wireless network150. The medical scan subsystems 101 can include a medical scan assistedreview system 102, medical scan report labeling system 104, a medicalscan annotator system 106, a medical scan diagnosing system 108, amedical scan interface feature evaluator system 110, a medical scanimage analysis system 112, a medical scan natural language analysissystem 114, and/or a medical scan comparison system 116. Some or all ofthe subsystems 101 can utilize the same processing devices, memorydevices, and/or network interfaces, for example, running on a same setof shared servers connected to network 150. Alternatively or inaddition, some or all of the subsystems 101 be assigned their ownprocessing devices, memory devices, and/or network interfaces, forexample, running separately on different sets of servers connected tonetwork 150. Some or all of the subsystems 101 can interact directlywith each other, for example, where one subsystem's output istransmitted directly as input to another subsystem via network 150.Network 150 can include one or more wireless and/or wired communicationsystems; one or more non-public intranet systems and/or public internetsystems; and/or one or more local area networks (LAN) and/or wide areanetworks (WAN).

The medical scan processing system 100 can further include a databasestorage system 140, which can include one or more servers, one or morememory devices of one or more subsystems 101, and/or one or more othermemory devices connected to network 150. The database storage system 140can store one or more shared databases and/or one or more files storedon one or more memory devices that include database entries as describedherein. The shared databases and/or files can each be utilized by someor all of the subsystems of the medical scan processing system, allowingsome or all of the subsystems and/or client devices to retrieve, edit,add, or delete entries to the one or more databases and/or files.

The one or more client devices 120 can each be associated with one ormore users of one or more subsystems of the medical scan processingsystem. Some or all of the client devices can be associated withhospitals or other medical institutions and/or associated with medicalprofessionals, employees, or other individual users for example, locatedat one or more of the medical institutions. Some of the client devices120 can correspond to one or more administrators of one or moresubsystems of the medical scan processing system, allowingadministrators to manage, supervise, or override functions of one ormore subsystems for which they are responsible.

Some or all of the subsystems 101 of the medical scan processing system100 can include a server that presents a website for operation via abrowser of client devices 120. Alternatively or in addition, each clientdevice can store application data corresponding to some or allsubsystems, for example, a subset of the subsystems that are relevant tothe user in a memory of the client device, and a processor of the clientdevice can display the interactive interface based on instructions inthe interface data stored in memory. For example, the website presentedby a subsystem can operate via the application. Some or all of thewebsites presented can correspond to multiple subsystems, for example,where the multiple subsystems share the server presenting the website.Furthermore, the network 150 can be configured for secure and/orauthenticated communications between the medical scan subsystems 101,the client devices 120 and the database storage system 140 to protectthe data stored in the database storage system and the data communicatedbetween the medical scan subsystems 101, the client devices 120 and thedatabase storage system 140 from unauthorized access.

The medical scan assisted review system 102 can be used to aid medicalprofessionals or other users in diagnosing, triaging, classifying,ranking, and/or otherwise reviewing medical scans by presenting amedical scan for review by a user by transmitting medical scan data of aselected medical scan and/or interface feature data of selectedinterface features of to a client device 120 corresponding to a user ofthe medical scan assisted review system for display via a display deviceof the client device. The medical scan assisted review system 102 cangenerate scan review data for a medical scan based on user input to theinteractive interface displayed by the display device in response toprompts to provide the scan review data, for example, where the promptscorrespond to one or more interface features.

The medical scan assisted review system 102 can be operable to receive,via a network, a medical scan for review. Abnormality annotation datacan be generated by identifying one or more of abnormalities in themedical scan by utilizing a computer vision model that is trained on aplurality of training medical scans. The abnormality annotation data caninclude location data and classification data for each of the pluralityof abnormalities and/or data that facilitates the visualization of theabnormalities in the scan image data. Report data including textdescribing each of the plurality of abnormalities is generated based onthe abnormality data. The visualization and the report data, which cancollectively be displayed annotation data, can be transmitted to aclient device. A display device associated with the client device candisplay the visualization in conjunction with the medical scan via aninteractive interface, and the display device can further display thereport data via the interactive interface.

In various embodiments, longitudinal data, such as one or moreadditional scans of longitudinal data 433 of the medical scan or ofsimilar scans, can be displayed in conjunction with the medical scanautomatically, or in response to the user electing to view longitudinaldata via user input. For example, the medical scan assisted reviewsystem can retrieve a previous scan or a future scan for the patientfrom a patient database or from the medical scan database automaticallyor in response to the user electing to view past patient data. One ormore previous scans can be displayed in one or more correspondingwindows adjacent to the current medical scan. For example, the user canselect a past scan from the longitudinal data for display. Alternativelyor in addition, the user can elect longitudinal parameters such asamount of time elapsed, scan type, electing to select the most recentand/or least recent scan, electing to select a future scan, electing toselect a scan at a date closest to the scan, or other criteria, and themedical scan assisted review system can automatically select a previousscan that compares most favorably to the longitudinal parameters. Theselected additional scan can be displayed in an adjacent windowalongside the current medical scan. In some embodiments, multipleadditional scans will be selected and can be displayed in multipleadjacent windows.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a selected additional scan will display image slices 412 determinedto correspond with the currently displayed slice 412 of the medicalscan. As described with respect to selecting a slice of a selectedsimilar medical scan for display, this can be achieved based onselecting the image slice with a matching slice number, based onautomatically determining the image slice that most closely matches theanatomical region corresponding to the currently displayed slice of thecurrent scan, and/or based on determining the slice in the previous scanwith the most similar view of the abnormality as the currently displayedslice. The user can use a single scroll bar or other single user inputindication to jump to a different image slice, and the multiple windowscan simultaneously display the same numbered image slice, or can scrollor jump by the same number of slices if different slice numbers areinitially displayed. In some embodiments, three or more adjacent windowscorresponding to the medical scan and two or more additional scans aredisplayed, and can all be controlled with the single scroll bar in asimilar fashion.

The medical scan assisted review system 102 can automatically detectprevious states of the identified abnormalities based on the abnormalitydata, such as the abnormality location data. The detected previousstates of the identified abnormality can be circled, highlighted, orotherwise indicated in their corresponding window. The medical scanassisted review system 102 can retrieve classification data for theprevious state of the abnormality by retrieving abnormality annotationdata 442 of the similar abnormality mapped to the previous scan from themedical scan database 342. This data may not be assigned to the previousscan, and the medical scan assisted review system can automaticallydetermine classification or other diagnosis data for the previousmedical scan by utilizing the medical scan image analysis system asdiscussed. Alternatively or in addition, some or all of the abnormalityclassification data 445 or other diagnosis data 440 for the previousscan can be assigned values determined based on the abnormalityclassification data or other diagnosis data determined for the currentscan. Such abnormality classification data 445 or other diagnosis data440 determined for the previous scan can be mapped to the previous scan,and or mapped to the longitudinal data 433, in the database and/ortransmitted to a responsible entity via the network.

The medical assisted review system can automatically generate statechange data such as a change in size, volume, malignancy, or otherchanges to various classifiers of the abnormality. This can be achievedby automatically comparing image data of one or more previous scans andthe current scan and/or by comparing abnormality data of the previousscan to abnormality data of the current scan. In some embodiments, suchmetrics can be calculated by utilizing the medical scan similarityanalysis function, for example, where the output of the medical scansimilarity analysis function such as the similarity score indicatesdistance, error, or other measured discrepancy in one or moreabnormality classifier categories 444 and/or abnormality patterncategories 446. This calculated distance, error, or other measureddiscrepancy in each category can be used to quantify state change data,indicate a new classifier in one or more categories, to determine if acertain category has become more or less severe, or otherwise determinehow the abnormality has changed over time. In various embodiments, thisdata can be displayed in one window, for example, where an increase inabnormality size is indicated by overlaying or highlighting an outlineof the current abnormality over the corresponding image slice of theprevious abnormality, or vice versa. In various embodiments whereseveral past scans are available, such state change data can bedetermined over time, and statistical data showing growth rate changesover time or malignancy changes over time can be generated, for example,indicating if a growth rate is lessening or worsening over time. Imageslices corresponding to multiple past scans can be displayed insequence, for example, where a first scroll bar allows a user to scrollbetween image slice numbers, and a second scroll bar allows a user toscroll between the same image slice over time. In various embodimentsthe abnormality data, heat map data, or other interface features will bedisplayed in conjunction with the image slices of the past image data.

The medical scan report labeling system 104 can be used to automaticallyassign medical codes to medical scans based on user identified keywords,phrases, or other relevant medical condition terms of natural text datain a medical scan report of the medical scan, identified by users of themedical scan report labeling system 104. The medical scan reportlabeling system 104 can be operable to transmit a medical report thatincludes natural language text to a first client device for display.Identified medical condition term data can be received from the firstclient device in response. An alias mapping pair in a medical labelalias database can be identified by determining that a medical conditionterm of the alias mapping pair compares favorably to the identifiedmedical condition term data. A medical code that corresponds to thealias mapping pair and a medical scan that corresponds to the medicalreport can be transmitted to a second client device of an expert userfor display, and accuracy data can be received from the second clientdevice in response. The medical code is mapped to the first medical scanin a medical scan database when the accuracy data indicates that themedical code compares favorably to the medical scan.

The medical scan annotator system 106 can be used to gather annotationsof medical scans based on review of the medical scan image data by usersof the system such as radiologists or other medical professionals.Medical scans that require annotation, for example, that have beentriaged from a hospital or other triaging entity, can be sent tomultiple users selected by the medical scan annotator system 106, andthe annotations received from the multiple medical professionals can beprocessed automatically by a processing system of the medical scanannotator system, allowing the medical scan annotator system toautomatically determine a consensus annotation of each medical scan.Furthermore, the users can be automatically scored by the medical scanannotator system based on how closely their annotation matches to theconsensus annotation or some other truth annotation, for example,corresponding to annotations of the medical scan assigned a truth flag.Users can be assigned automatically to annotate subsequent incomingmedical scans based on their overall scores and/or based on categorizedscores that correspond to an identified category of the incoming medicalscan.

The medical scan annotator system 106 can be operable to select amedical scan for transmission via a network to a first client device anda second client device for display via an interactive interface, andannotation data can be received from the first client device and thesecond client device in response. Annotation similarity data can begenerated by comparing the first annotation data to the secondannotation data, and consensus annotation data can be generated based onthe first annotation data and the second annotation data in response tothe annotation similarity data indicating that the difference betweenthe first annotation data and the second annotation data comparesfavorably to an annotation discrepancy threshold. The consensusannotation data can be mapped to the medical scan in a medical scandatabase.

A medical scan diagnosing system 108 can be used by hospitals, medicalprofessionals, or other medical entities to automatically produceinference data for given medical scans by utilizing computer visiontechniques and/or natural language processing techniques. Thisautomatically generated inference data can be used to generate and/orupdate diagnosis data or other corresponding data of correspondingmedical scan entries in a medical scan database. The medical scandiagnosing system can utilize a medical scan database, user database,and/or a medical scan analysis function database by communicating withthe database storage system 140 via the network 150, and/or can utilizeanother medical scan database, user database, and/or function databasestored in local memory.

The medical scan diagnosing system 108 can be operable to receive amedical scan. Diagnosis data of the medical scan can be generated byperforming a medical scan inference function on the medical scan. Thefirst medical scan can be transmitted to a first client deviceassociated with a user of the medical scan diagnosing system in responseto the diagnosis data indicating that the medical scan corresponds to anon-normal diagnosis. The medical scan can be displayed to the user viaan interactive interface displayed by a display device corresponding tothe first client device. Review data can be received from the firstclient device, where the review data is generated by the first clientdevice in response to a prompt via the interactive interface. Updateddiagnosis data can be generated based on the review data. The updateddiagnosis data can be transmitted to a second client device associatedwith a requesting entity.

A medical scan interface feature evaluating system 110 can be usedevaluate proposed interface features or currently used interfacefeatures of an interactive interface to present medical scans for reviewby medical professionals or other users of one or more subsystems 101.The medical scan interface feature evaluator system 110 can be operableto generate an ordered image-to-prompt mapping by selecting a set ofuser interface features to be displayed with each of an ordered set ofmedical scans. The set of medical scans and the ordered image-to-promptmapping can be transmitted to a set of client devices. A set ofresponses can be generated by each client device in response tosequentially displaying each of the set of medical scans in conjunctionwith a mapped user interface feature indicated in the orderedimage-to-prompt mapping via a user interface. Response score data can begenerated by comparing each response to truth annotation data of thecorresponding medical scan. Interface feature score data correspondingto each user interface feature can be generated based on aggregating theresponse score data, and is used to generate a ranking of the set ofuser interface features.

A medical scan image analysis system 112 can be used to generate and/orperform one or more medical scan image analysis functions by utilizing acomputer vision-based learning algorithm 1350 on a training set ofmedical scans with known annotation data, diagnosis data, labelingand/or medical code data, report data, patient history data, patientrisk factor data, and/or other metadata associated with medical scans.These medical scan image analysis functions can be used to generateinference data for new medical scans that are triaged or otherwiserequire inferred annotation data, diagnosis data, labeling and/ormedical code data, and/or report data. For example, some medical scanimage analysis functions can correspond to medical scan inferencefunctions of the medical scan diagnosing system or other medical scananalysis functions of a medical scan analysis function database. Themedical scan image analysis functions can be used to determine whetheror not a medical scan is normal, to detect the location of anabnormality in one or more slices of a medical scan, and/or tocharacterize a detected abnormality. The medical scan image analysissystem can be used to generate and/or perform computer vision basedmedical scan image analysis functions utilized by other subsystems ofthe medical scan processing system as described herein, aiding medicalprofessionals to diagnose patients and/or to generate further data andmodels to characterize medical scans. The medical scan image analysissystem can include a processing system that includes a processor and amemory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations.

The medical scan image analysis system 112 can be operable to receive aplurality of medical scans that represent a three-dimensional anatomicalregion and include a plurality of cross-sectional image slices. Aplurality of three-dimensional subregions corresponding to each of theplurality of medical scans can be generated by selecting a proper subsetof the plurality of cross-sectional image slices from each medical scan,and by further selecting a two-dimensional subregion from each propersubset of cross-sectional image slices. A learning algorithm can beperformed on the plurality of three-dimensional subregions to generate aneural network. Inference data corresponding to a new medical scanreceived via the network can be generated by performing an inferencealgorithm on the new medical scan by utilizing the neural network. Aninferred abnormality can be identified in the new medical scan based onthe inference data.

The medical scan natural language analysis system 114 can determine atraining set of medical scans with medical codes determined to be truthdata. Corresponding medical reports and/or other natural language textdata associated with a medical scan can be utilized to train a medicalscan natural language analysis function by generating a medical reportnatural language model. The medical scan natural language analysisfunction can be utilized to generate inference data for incoming medicalreports for other medical scans to automatically determine correspondingmedical codes, which can be mapped to corresponding medical scans.Medical codes assigned to medical scans by utilizing the medical reportnatural language model can be utilized by other subsystems, for example,to train other medical scan analysis functions, to be used as truth datato verify annotations provided via other subsystems, to aid indiagnosis, or otherwise be used by other subsystems as described herein.

A medical scan comparison system 116 can be utilized by one or moresubsystems to identify and/or display similar medical scans, forexample, to perform or determine function parameters for a medical scansimilarity analysis function, to generate or retrieve similar scan data,or otherwise compare medical scan data. The medical scan comparisonsystem 116 can also utilize some or all features of other subsystems asdescribed herein. The medical scan comparison system 116 can be operableto receive a medical scan via a network and can generate similar scandata. The similar scan data can include a subset of medical scans from amedical scan database and can be generated by performing an abnormalitysimilarity function, such as medical scan similarity analysis function,to determine that a set of abnormalities included in the subset ofmedical scans compare favorably to an abnormality identified in themedical scan. At least one cross-sectional image can be selected fromeach medical scan of the subset of medical scans for display on adisplay device associated with a user of the medical scan comparisonsystem in conjunction with the medical scan.

FIG. 2A presents an embodiment of client device 120. Each client device120 can include one or more client processing devices 230, one or moreclient memory devices 240, one or more client input devices 250, one ormore client network interfaces 260 operable to more support one or morecommunication links via the network 150 indirectly and/or directly,and/or one or more client display devices 270, connected via bus 280.Client applications 202, 204, 206, 208, 210, 212, 214, and/or 216correspond to subsystems 102, 104, 106, 108, 110, 112, 114, and/or 116of the medical scan processing system respectfully. Each client device120 can receive the application data from the corresponding subsystemvia network 150 by utilizing network interface 260, for storage in theone or more memory devices 240. In various embodiments, some or allclient devices 120 can include a computing device associated with aradiologist, medical entity, or other user of one or more subsystems asdescribed herein.

The one or more processing devices 230 can display interactive interface275 on the one or more client display devices 270 in accordance with oneor more of the client applications 202, 204, 206, 208, 210, 212, 214,and/or 216, for example, where a different interactive interface 275 isdisplayed for some or all of the client applications in accordance withthe website presented by the corresponding subsystem 102, 104, 106, 108,110, 112, 114 and/or 116. The user can provide input in response to menudata or other prompts presented by the interactive interface via the oneor more client input devices 250, which can include a microphone, mouse,keyboard, touchscreen of display device 270 itself or other touchscreen,and/or other device allowing the user to interact with the interactiveinterface. The one or more processing devices 230 can process the inputdata and/or send raw or processed input data to the correspondingsubsystem, and/or can receive and/or generate new data in response forpresentation via the interactive interface 275 accordingly, by utilizingnetwork interface 260 to communicate bidirectionally with one or moresubsystems and/or databases of the medical scan processing system vianetwork 150.

FIG. 2B presents an embodiment of a subsystem 101, which can be utilizedin conjunction with subsystem 102, 104, 106, 108, 110, 112, 114 and/or116. Each subsystem 101 can include one or more subsystem processingdevices 235, one or more subsystem memory devices 245, and/or one ormore subsystem network interfaces 265, connected via bus 285. Thesubsystem memory devices 245 can store executable instructions that,when executed by the one or more subsystem processing devices 235,facilitate performance of operations by the subsystem 101, as describedfor each subsystem herein.

FIG. 3 presents an embodiment of the database storage system 140.Database storage system 140 can include at least one database processingdevice 330, at least one database memory device 340, and at least onedatabase network interface 360, operable to more support one or morecommunication links via the network 150 indirectly and/or directly, allconnected via bus 380. The database storage system 140 can store one ormore databases the at least one memory 340, which can include a medicalscan database 342 that includes a plurality medical scan entries 352, auser database 344 that includes a plurality of user profile entries 354,a medical scan analysis function database 346 that includes a pluralityof medical scan analysis function entries 356, an interface featuredatabase 348 can include a plurality of interface feature entries 358,and/or other databases that store data generated and/or utilized by thesubsystems 101. Some or all of the databases 342, 344, 346 and/or 348can consist of multiple databases, can be stored relationally ornon-relationally, and can include different types of entries anddifferent mappings than those described herein. A database entry caninclude an entry in a relational table or entry in a non-relationalstructure. Some or all of the data attributes of an entry 352, 354, 356,and/or 358 can refer to data included in the entry itself or that isotherwise mapped to an identifier included in the entry and can beretrieved from, added to, modified, or deleted from the database storagesystem 140 based on a given identifier of the entry. Some or all of thedatabases 342, 344, 346, and/or 348 can instead be stored locally by acorresponding subsystem, for example, if they are utilized by only onesubsystem.

The processing device 330 can facilitate read/write requests receivedfrom subsystems and/or client devices via the network 150 based onread/write permissions for each database stored in the at least onememory device 340. Different subsystems can be assigned differentread/write permissions for each database based on the functions of thesubsystem, and different client devices 120 can be assigned differentread/write permissions for each database. One or more client devices 120can correspond to one or more administrators of one or more of thedatabases stored by the database storage system, and databaseadministrator devices can manage one or more assigned databases,supervise assess and/or efficiency, edit permissions, or otherwiseoversee database processes based on input to the client device viainteractive interface 275.

FIG. 4A presents an embodiment of a medical scan entry 352, stored inmedical scan database 342, included in metadata of a medical scan,and/or otherwise associated with a medical scan. A medical scan caninclude imaging data corresponding to a CT scan, x-ray, MM, PET scan,Ultrasound, EEG, mammogram, or other type of radiological scan ormedical scan taken of an anatomical region of a human body, animal,organism, or object and further can include metadata corresponding tothe imaging data. Some or all of the medical scan entries can beformatted in accordance with a Digital Imaging and Communications inMedicine (DICOM) format or other standardized image format, and some ormore of the fields of the medical scan entry 352 can be included in aDICOM header or other standardized header of the medical scan. Medicalscans can be awaiting review or can have already been reviewed by one ormore users or automatic processes and can include tentative diagnosisdata automatically generated by a subsystem, generated based on userinput, and/or generated from another source. Some medical scans caninclude final, known diagnosis data generated by a subsystem and/orgenerated based on user input, and/or generated from another source, andcan included in training sets used to train processes used by one ormore subsystems such as the medical scan image analysis system 112and/or the medical scan natural language analysis system 114.

Some medical scans can include one or more abnormalities, which can beidentified by a user or can be identified automatically. Abnormalitiescan include nodules, for example malignant nodules identified in a chestCT scan. Abnormalities can also include and/or be characterized by oneor more abnormality pattern categories such as such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, for exampleidentified in a chest x-ray. Abnormalities can also include any otherunknown, malignant or benign feature of a medical scan identified as notnormal. Some scans can contain zero abnormalities, and can be identifiedas normal scans. Some scans identified as normal scans can includeidentified abnormalities that are classified as benign, and include zeroabnormalities classified as either unknown or malignant. Scansidentified as normal scans may include abnormalities that were notdetected by one or more subsystems and/or by an originating entity.Thus, some scans may be improperly identified as normal. Similarly,scans identified to include at least one abnormality may include atleast one abnormality that was improperly detected as an abnormality byone or more subsystems and/or by an originating entity. Thus, some scansmay be improperly identified as containing abnormalities.

Each medical scan entry 352 can be identified by its own medical scanidentifier 353, and can include or otherwise map to medical scan imagedata 410, and metadata such as scan classifier data 420, patient historydata 430, diagnosis data 440, annotation author data 450, confidencescore data 460, display parameter data 470, similar scan data 480,training set data 490, and/or other data relating to the medical scan.Some or all of the data included in a medical scan entry 352 can be usedto aid a user in generating or editing diagnosis data 440, for example,in conjunction with the medical scan assisted review system 102, themedical scan report labeling system 104, and/or the medical scanannotator system 106. Some or all of the data included in a medical scanentry 352 can be used to allow one or more subsystems 101, such asautomated portions of the medical scan report labeling system 104 and/orthe medical scan diagnosing system 108, to automatically generate and/oredit diagnosis data 440 or other data the medical scan. Some or all ofthe data included in a medical scan entry 352 can be used to train someor all medical scan analysis functions of the medical scan analysisfunction database 346 such as one or more medical scan image analysisfunctions, one or more medical scan natural language analysis functions,one or more medical scan similarity analysis functions, one or moremedical report generator functions, and/or one or more medical reportanalysis functions, for example, in conjunction with the medical scanimage analysis system 112, the medical scan natural language analysissystem 114, and/or the medical scan comparison system 116.

The medical scan entries 352 and the associated data as described hereincan also refer to data associated with a medical scan that is not storedby the medical scan database, for example, that is uploaded by a clientdevice for direct transmission to a subsystem, data generated by asubsystem and used as input to another subsystem or transmitted directlyto a client device, data stored by a Picture Archive and CommunicationSystem (PACS) communicating with the medical scan processing system 100,or other data associated with a medical scan that is received and orgenerated without being stored in the medical scan database 342. Forexample, some or all of the structure and data attributes described withrespect to a medical scan entry 352 can also correspond to structureand/or data attribute of data objects or other data generated by and/ortransmitted between subsystems and/or client devices that correspond toa medical scan. Herein, any of the data attributes described withrespect to a medical scan entry 352 can also correspond to dataextracted from a data object generated by a subsystem or client deviceor data otherwise received from a subsystem, client device, or othersource via network 150 that corresponds to a medical scan.

The medical scan image data 410 can include one or more imagescorresponding to a medical scan. The medical scan image data 410 caninclude one or more image slices 412, for example, corresponding to asingle x-ray image, a plurality of cross-sectional, tomographic imagesof a scan such as a CT scan, or any plurality of images taken from thesame or different point at the same or different angles. The medicalscan image data 410 can also indicate an ordering of the one or moreimage slices 412. Herein, a “medical scan” can refer a full scan of anytype represented by medical scan image data 410. Herein, an “imageslice” can refer to one of a plurality of cross-sectional images of themedical scan image data 410, one of a plurality of images taken fromdifferent angles of the medical scan image data 410, and/or the singleimage of the medical scan image data 410 that includes only one image.Furthermore “plurality of image slices” can refer to all of the imagesof the associated medical scan, and refers to only a single image if themedical scan image data 410 includes only one image. Each image slice412 can include a plurality of pixel values 414 mapped to each pixel ofthe image slice. Each pixel value can correspond to a density value,such as a Hounsfield value or other measure of density. Pixel values canalso correspond to a grayscale value, a RGB (Red-Green-Blue) or othercolor value, or other data stored by each pixel of an image slice 412.

Scan classifier data 420 can indicate classifying data of the medicalscan. Scan classifier data can include scan type data 421, for example,indicating the modality of the scan. The scan classifier data canindicate that the scan is a CT scan, x-ray, MM, PET scan, Ultrasound,EEG, mammogram, or other type of scan. Scan classifier data 420 can alsoinclude anatomical region data 422, indicating for example, the scan isa scan of the chest, head, right knee, or other anatomical region. Scanclassifier data can also include originating entity data 423, indicatingthe hospital where the scan was taken and/or a user that uploaded thescan to the system. If the originating entity data corresponds to a userof one or more subsystems 101, the originating entity data can include acorresponding user profile identifier and/or include other data from theuser profile entry 354 of the user. Scan classifier data 420 can includegeographic region data 424, indicating a city, state, and/or countryfrom which the scan originated, for example, based on the user dataretrieved from the user database 344 based on the originating entity.Scan classifier data can also include machine data 425, which caninclude machine identifier data, machine model data, machine calibrationdata, and/or contrast agent data, for example based on imaging machinedata retrieved from the user database 344 based on the originatingentity data 423. The scan classifier data 420 can include scan date data426 indicating when the scan was taken. The scan classifier data 420 caninclude scan priority data 427, which can indicate a priority score,ranking, number in a queue, or other priority data with regard totriaging and/or review. A priority score, ranking, or queue number ofthe scan priority data 427 can be generated by automatically by asubsystem based on the scan priority data 427, based on a severity ofpatient symptoms or other indicators in the risk factor data 432, basedon a priority corresponding to the originating entity, based onpreviously generated diagnosis data 440 for the scan, and/or can beassigned by the originating entity and/or a user of the system.

The scan classifier data 420 can include other classifying data notpictured in FIG. 4A. For example, a set of scans can include medicalscan image data 410 corresponding to different imaging planes. The scanclassifier data can further include imaging plane data indicating one ormore imaging planes corresponding to the image data. For example, theimaging plane data can indicate the scan corresponds to the axial plane,sagittal plane, or coronal plane. A single medical scan entry 352 caninclude medical scan image data 410 corresponding multiple planes, andeach of these planes can be tagged appropriately in the image data. Inother embodiments, medical scan image data 410 corresponding to eachplane can be stored as separate medical scan entries 352, for example,with a common identifier indicating these entries belong to the same setof scans.

Alternatively or in addition, the scan classifier data 420 can includesequencing data. For example, a set of scans can include medical scanimage data 410 corresponding to different sequences. The scan classifierdata can further include sequencing data indicating one or more of aplurality of sequences of the image data corresponds to, for example,indicating whether an MRI scan corresponds to a T2 sequence, a T1sequence, a T1 sequence with contrast, a diffusion sequence, a FLAIRsequence, or other MRI sequence. A single medical scan entry 352 caninclude medical scan image data 410 corresponding to multiple sequences,and each of these sequences can be tagged appropriately in the entry. Inother embodiments, medical scan image data 410 corresponding to eachsequence can be stored as separate medical scan entries 352, forexample, with a common identifier indicating these entries belong to thesame set of scans.

Alternatively or in addition, the scan classifier data 420 can includean image quality score. This score can be determined automatically byone or more subsystems 101, and/or can be manually assigned the medicalscan. The image quality score can be based on a resolution of the imagedata 410, where higher resolution image data is assigned a morefavorable image quality score than lower resolution image data. Theimage quality score can be based on whether the image data 410corresponds to digitized image data received directly from thecorresponding imaging machine, or corresponds to a hard copy of theimage data that was later scanned in. In some embodiments, the imagequality score can be based on a detected corruption, and/or detectedexternal factor that determined to negatively affect the quality of theimage data during the capturing of the medical scan and/or subsequent tothe capturing of the medical scan. In some embodiments, the imagequality score can be based on detected noise in the image data, where amedical scan with a higher level of detected noise can receive a lessfavorable image quality score than a medical scan with a lower level ofdetected noise. Medical scans with this determined corruption orexternal factor can receive a less favorable image quality score thanmedical scans with no detected corruption or external factor.

In some embodiments, the image quality score can be based on includemachine data 425. In some embodiments, one or more subsystems canutilize the image quality score to flag medical scans with image qualityscores that fall below an image quality threshold. The image qualitythreshold can be the same or different for different subsystems, medicalscan modalities, and/or anatomical regions. For example, the medicalscan image analysis system can automatically filter training sets basedon selecting only medical scans with image quality scores that comparefavorably to the image quality threshold. As another example, one ormore subsystems can flag a particular imaging machine and/or hospital orother medical entity that have produced at least a threshold numberand/or percentage of medical scan with image quality scores that compareunfavorably to the image quality threshold. As another example, ade-noising algorithm can be automatically utilized to clean the imagedata when the image quality score compares unfavorably to the imagequality threshold. As another example, the medical scan image analysissystem can select a particular medical image analysis function from aset of medical image analysis functions to utilize on a medical scan togenerate inference data for the medical scan. Each of this set ofmedical image analysis function can be trained on different levels ofimage quality, and the selected image analysis function can be selectedbased on the determined image quality score falling within a range ofimage quality scores the image analysis function was trained on and/oris otherwise suitable for.

The patient history data 430 can include patient identifier data 431which can include basic patient information such as name or anidentifier that may be anonymized to protect the confidentiality of thepatient, age, and/or gender. The patient identifier data 431 can alsomap to a patient entry in a separate patient database stored by thedatabase storage system, or stored elsewhere. The patient history datacan include patient risk factor data 432 which can include previousmedical history, family medical history, smoking and/or drug habits,pack years corresponding to tobacco use, environmental exposures,patient symptoms, etc. The patient history data 430 can also includelongitudinal data 433, which can identify one or more additional medicalscans corresponding to the patient, for example, retrieved based onpatient identifier data 431 or otherwise mapped to the patientidentifier data 431. Some or all additional medical scans can beincluded in the medical scan database, and can be identified based ontheir corresponding identifiers medical scan identifiers 353. Some orall additional medical scans can be received from a different source andcan otherwise be identified. Alternatively or in addition, thelongitudinal data can simply include some or all relevant scan entrydata of a medical scan entry 352 corresponding to the one or moreadditional medical scans. The additional medical scans can be the sametype of scan or different types of scans. Some or all of the additionalscans may correspond to past medical scans, and/or some or all of theadditional scans may correspond to future medical scans. Thelongitudinal data 433 can also include data received and/or determinedat a date after the scan such as final biopsy data, or some or all ofthe diagnosis data 440. The patient history data can also include alongitudinal quality score 434, which can be calculated automatically bya subsystem, for example, based on the number of additional medicalscans, based on how many of the additional scans in the file were takenbefore and/or after the scan based on the scan date data 426 of themedical scan and the additional medical scans, based on a date rangecorresponding to the earliest scan and corresponding to the latest scan,based on the scan types data 421 these scans, and/or based on whether ornot a biopsy or other final data is included. As used herein, a “high”longitudinal quality score refers to a scan having more favorablelongitudinal data than that with a “low” longitudinal quality score.

Diagnosis data 440 can include data that indicates an automateddiagnosis, a tentative diagnosis, and/or data that can otherwise be usedto support medical diagnosis, triage, medical evaluation and/or otherreview by a medical professional or other user. The diagnosis data 440of a medical scan can include a binary abnormality identifier 441indicating whether the scan is normal or includes at least oneabnormality. In some embodiments, the binary abnormality identifier 441can be determined by comparing some or all of confidence score data 460to a threshold, can be determined by comparing a probability value to athreshold, and/or can be determined by comparing another continuous ordiscrete value indicating a calculated likelihood that the scan containsone or more abnormalities to a threshold. In some embodiments,non-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the scan contains one or moreabnormalities, can be included in diagnosis data 440 in addition to, orinstead of, binary abnormality identifier 441. One or abnormalities canbe identified by the diagnosis data 440, and each identified abnormalitycan include its own set of abnormality annotation data 442.Alternatively, some or all of the diagnosis data 440 can indicate and/ordescribe multiple abnormalities, and thus will not be presented for eachabnormality in the abnormality annotation data 442. For example, thereport data 449 of the diagnosis data 440 can describe all identifiedabnormalities, and thus a single report can be included in thediagnosis.

FIG. 4B presents an embodiment of the abnormality annotation data 442.The abnormality annotation data 442 for each abnormality can includeabnormality location data 443, which can include an anatomical locationand/or a location specific to pixels, image slices, coordinates or otherlocation information identifying regions of the medical scan itself. Theabnormality annotation data 442 can include abnormality classificationdata 445 which can include binary, quantitative, and/or descriptive dataof the abnormality as a whole, or can correspond to one or moreabnormality classifier categories 444, which can include size, volume,pre-post contrast, doubling time, calcification, components, smoothness,spiculation, lobulation, sphericity, internal structure, texture, orother categories that can classify and/or otherwise characterize anabnormality. Abnormality classifier categories 444 can be assigned abinary value, indicating whether or not such a category is present. Forexample, this binary value can be determined by comparing some or all ofconfidence score data 460 to a threshold, can be determined by comparinga probability value to a threshold, and/or can be determined bycomparing another continuous or discrete value indicating a calculatedlikelihood that a corresponding abnormality classifier category 444 ispresent to a threshold, which can be the same or different threshold foreach abnormality classifier category 444. In some embodiments,abnormality classifier categories 444 can be assigned one or morenon-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the corresponding classifier category 444is present.

The abnormality classifier categories 444 can also include a malignancycategory, and the abnormality classification data 445 can include amalignancy rating such as a Lung-RADS score, a Fleischner score, and/orone or more calculated values that indicate malignancy level, malignancyseverity, and/or probability of malignancy. Alternatively or inaddition, the malignancy category can be assigned a value of “yes”,“no”, or “maybe”. The abnormality classifier categories 444 can alsoinclude abnormality pattern categories 446 such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, and the abnormalityclassification data 445 for each abnormality pattern category 446 canindicate whether or not each of the abnormality patterns is present.

The abnormality classifier categories can correspond to ResponseEvaluation Criteria in Solid Tumors (RECIST) eligibility and/or RECISTevaluation categories. For example, an abnormality classifier category444 corresponding to RECIST eligibility can have correspondingabnormality classification data 445 indicating a binary value “yes” or“no”, and/or can indicate if the abnormality is a “target lesion” and/ora “non-target lesion.” As another example, an abnormality classifiercategory 444 corresponding to a RECIST evaluation category can bedetermined based on longitudinal data 433 and can have correspondingabnormality classification data 445 that includes one of the set ofpossible values “Complete Response”, “Partial Response”, “StableDisease”, or “Progressive Disease.”

The diagnosis data 440 as a whole, and/or the abnormality annotationdata 442 for each abnormality, can include custom codes or datatypesidentifying the binary abnormality identifier 441, abnormality locationdata 443 and/or some or all of the abnormality classification data 445of one or more abnormality classifier categories 444. Alternatively orin addition, some or all of the abnormality annotation data 442 for eachabnormality and/or other diagnosis data 440 can be presented in a DICOMformat or other standardized image annotation format, and/or can beextracted into custom datatypes based on abnormality annotation dataoriginally presented in DICOM format. Alternatively or in addition, thediagnosis data 440 and/or the abnormality annotation data 442 for eachabnormality can be presented as one or more medical codes 447 such asSNOMED codes, Current Procedure Technology (CPT) codes, ICD-9 codes,ICD-10 codes, or other standardized medical codes used to label orotherwise describe medical scans.

Alternatively or in addition, the diagnosis data 440 can include naturallanguage text data 448 annotating or otherwise describing the medicalscan as a whole, and/or the abnormality annotation data 442 can includenatural language text data 448 annotating or otherwise describing eachcorresponding abnormality. In some embodiments, some or all of thediagnosis data 440 is presented only as natural language text data 448.In some embodiments, some or all of the diagnosis data 440 isautomatically generated by one or more subsystems based on the naturallanguage text data 448, for example, without utilizing the medical scanimage data 410, for example, by utilizing one or more medical scannatural language analysis functions trained by the medical scan naturallanguage analysis system 114. Alternatively or in addition, someembodiments, some or all of the natural language text data 448 isgenerated automatically based on other diagnosis data 440 such asabnormality annotation data 442, for example, by utilizing a medicalscan natural language generating function trained by the medical scannatural language analysis system 114.

The diagnosis data can include report data 449 that includes at leastone medical report, which can be formatted to include some or all of themedical codes 447, some or all of the natural language text data 448,other diagnosis data 440, full or cropped images slices formatted basedon the display parameter data 470 and/or links thereto, full or croppedimages slices or other data based on similar scans of the similar scandata 480 and/or links thereto, full or cropped images or other databased on patient history data 430 such as longitudinal data 433 and/orlinks thereto, and/or other data or links to data describing the medicalscan and associated abnormalities. The diagnosis data 440 can alsoinclude finalized diagnosis data corresponding to future scans and/orfuture diagnosis for the patient, for example, biopsy data or otherlongitudinal data 433 determined subsequently after the scan. Themedical report of report data 449 can be formatted based on specifiedformatting parameters such as font, text size, header data, bulleting ornumbering type, margins, file type, preferences for including one ormore full or cropped image slices 412, preferences for including similarmedical scans, preferences for including additional medical scans, orother formatting to list natural language text data and/or image data,for example, based on preferences of a user indicated in the originatingentity data 423 or other responsible user in the corresponding reportformatting data.

Annotation author data 450 can be mapped to the diagnosis data for eachabnormality, and/or mapped to the scan as a whole. This can include oneor more annotation author identifiers 451, which can include one or moreuser profile identifiers of a user of the system, such as an individualmedical professional, medical facility and/or medical entity that usesthe system. Annotation author data 450 can be used to determine theusage data of a user profile entry 354. Annotation author data 450 canalso include one or more medical scan analysis function identifiers 357or other function identifier indicating one or more functions or otherprocesses of a subsystem responsible for automatically generating and/orassisting a user in generating some or all of the diagnosis data, forexample an identifier of a particular type and/or version of a medicalscan image analysis functions that was used by the medical scandiagnosing system 108 used to generate part or all of the diagnosis data440 and/or an interface feature identifier, indicating an one or moreinterface features presented to a user to facilitate entry of and/orreviewing of the diagnosis data 440. The annotation author data can alsosimply indicate, for one or more portions of the diagnosis data 440, ifthis portion was generated by a human or automatically generated by asubsystem of the medical scan processing system.

In some embodiments, if a medical scan was reviewed by multipleentities, multiple, separate diagnosis data entries 440 can be includedin the medical scan entry 352, mapped to each diagnosis author in theannotation author data 450. This allows different versions of diagnosisdata 440 received from multiple entities. For example, annotation authordata of a particular medical scan could indicate that the annotationdata was written by a doctor at medical entity A, and the medical codedata was generated by user Y by utilizing the medical scan reportlabeling system 104, which was confirmed by expert user X. Theannotation author data of another medical scan could indicate that themedical code was generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, andconfirmed by expert user X. The annotation author data of anothermedical scan could indicate that the location and a first malignancyrating were generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, and thata second malignancy rating was entered by user Z. In some embodiments,one of the multiple diagnosis entries can include consensus annotationdata, for example, generated automatically by a subsystem such as themedical scan annotating system 106 based on the multiple diagnosis data440, based on confidence score data 460 of each of the multiplediagnosis data 440, and/or based on performance score data of acorresponding user, a medical scan analysis function, or an interfacefeature, identified in the annotation author data for each correspondingone of the multiple diagnosis data 440.

Confidence score data 460 can be mapped to some or all of the diagnosisdata 440 for each abnormality, and/or for the scan as a whole. This caninclude an overall confidence score for the diagnosis, a confidencescore for the binary indicator of whether or not the scan was normal, aconfidence score for the location a detected abnormality, and/orconfidence scores for some or all of the abnormality classifier data.This may be generated automatically by a subsystem, for example, basedon the annotation author data and corresponding performance score of oneor more identified users and/or subsystem attributes such as interactiveinterface types or medical scan image analysis functions indicated bythe annotation author data. In the case where multiple diagnosis dataentries 440 are included from different sources, confidence score data460 can be computed for each entry and/or an overall confidence score,for example, corresponding to consensus diagnosis data, can be based oncalculated distance or other error and/or discrepancies between theentries, and/or can be weighted on the confidence score data 460 of eachentry. In various embodiments, the confidence score data 460 can includea truth flag 461 indicating the diagnosis data is considered as “known”or “truth”, for example, flagged based on user input, flaggedautomatically based on the author data, and/or flagged automaticallybased on the calculated confidence score of the confidence score dataexceeding a truth threshold. As used herein, a “high” confidence scorerefers to a greater degree or more favorable level of confidence than a“low” confidence score.

Display parameter data 470 can indicate parameters indicating an optimalor preferred display of the medical scan by an interactive interface 275and/or formatted report for each abnormality and/or for the scan as awhole. Some or all of the display parameter data can have separateentries for each abnormality, for example, generated automatically by asubsystem 101 based on the abnormality annotation data 442. Displayparameter data 470 can include interactive interface feature data 471,which can indicate one or more selected interface features associatedwith the display of abnormalities and/or display of the medical scan asa whole, and/or selected interface features associated with userinteraction with a medical scan, for example, based on categorizedinterface feature performance score data and a category associated withthe abnormality and/or with the medical scan itself. The displayparameter data can include a slice subset 472, which can indicate aselected subset of the plurality of image slices that includes a singleimage slice 412 or multiple image slices 412 of the medical scan imagedata 410 for display by a user interface. The display parameter data 470can include slice order data 473 that indicates a selected customordering and/or ranking for the slice subset 472, or for all of theslices 412 of the medical scan. The display parameter data 470 caninclude slice cropping data 474 corresponding to some or all of theslice subset 472, or all of the image slices 412 of the medical scan,and can indicating a selected custom cropped region of each image slice412 for display, or the same selected custom cropped region for theslice subset 472 or for all slices 412. The display parameter data caninclude density window data 475, which can indicate a selected customdensity window for display of the medical scan as a whole, a selectedcustom density window for the slice subset 472, and/or selected customdensity windows for each of the image slices 412 of the slice subset472, and/or for each image slice 412 of the medical scan. The densitywindow data 475 can indicate a selected upper density value cut off anda selected lower density value cut off, and/or can include a selecteddeterministic function to map each density value of a pixel to agrayscale value based on the preferred density window. The interactiveinterface feature data 471, slice subset 472, slice order data 473,slice cropping data 474, and/or the density window data 475 can beselected via user input and/or generated automatically by one or moresubsystems 101, for example, based on the abnormality annotation data442 and/or based on performance score data of different interactiveinterface versions.

Similar scan data 480 can be mapped to each abnormality, or the scan asa whole, and can include similar scan identifier data 481 correspondingto one or more identified similar medical scans, for example,automatically identified by a subsystem 101, for example, by applying asimilar scan identification step of the medical scan image analysissystem 112 and/or applying medical scan similarity analysis function tosome or all of the data stored in the medical scan entry of the medicalscan, and/or to some or all corresponding data of other medical scans inthe medical scan database. The similar scan data 480 can also correspondto medical scans received from another source. The stored similaritydata can be used to present similar cases to users of the system and/orcan be used to train medical scan image analysis functions or medicalscan similarity analysis functions.

Each identified similar medical scan can have its own medical scan entry352 in the medical scan database 342 with its own data, and the similarscan identifier data 481 can include the medical scan identifier 353each similar medical scan. Each identified similar medical scan can be ascan of the same scan type or different scan type than medical scan.

The similar scan data 480 can include a similarity score 482 for eachidentified similar scan, for example, generated based on some or all ofthe data of the medical scan entry 352 for medical scan and based onsome or all of the corresponding data of the medical scan entry 352 forthe identified similar medical scan. For example, the similarity score482 can be generated based on applying a medical scan similarityanalysis function to the medical image scan data of medical scans and402, to some or all of the abnormality annotation data of medical scansand 402, and/or to some or all of the patient history data 430 ofmedical scans and 402 such as risk factor data 432. As used herein, a“high” similarity score refers a higher level of similarity that a “low”similarity score.

The similar scan data 480 can include its own similar scan displayparameter data 483, which can be determined based on some or all of thedisplay parameter data 470 of the identified similar medical scan. Someor all of the similar scan display parameter data 483 can be generatedautomatically by a subsystem, for example, based on the displayparameter data 470 of the identified similar medical scan, based on theabnormality annotation data 442 of the medical scan itself and/or basedon display parameter data 470 of the medical scan itself. Thus, thesimilar scan display parameter data 483 can be the same or differentthan the display parameter data 470 mapped to the identified similarmedical scan and/or can be the same or different than the displayparameter data 470 of the medical scan itself. This can be utilized whendisplaying similar scans to a user via interactive interface 275 and/orcan be utilized when generating report data 449 that includes similarscans, for example, in conjunction with the medical scan assisted reviewsystem 102.

The similar scan data 480 can include similar scan abnormality data 484,which can indicate one of a plurality of abnormalities of the identifiedsimilar medical scan and its corresponding abnormality annotation data442. For example, the similarity scan abnormality data 484 can includean abnormality pair that indicates one of a plurality of abnormalitiesof the medical scan, and indicates one of a plurality of abnormalitiesof the identified similar medical scan, for example, that was identifiedas the similar abnormality.

The similar scan data 480 can include similar scan filter data 485. Thesimilar scan filter data can be generated automatically by a subsystem,and can include a selected ordered or un-ordered subset of allidentified similar scans of the similar scan data 480, and/or a rankingof all identified similar scans. For example, the subset can be selectedand/or some or all identified similar scans can be ranked based on eachsimilarity score 482, and/or based on other factors such as based on alongitudinal quality score 434 of each identified similar medical scan.

The training set data 490 can indicate one or more training sets thatthe medical scan belongs to. For example, the training set data canindicate one or more training set identifiers 491 indicating one or moremedical scan analysis functions that utilized the medical scan in theirtraining set, and/or indicating a particular version identifier 641 ofthe one or more medical scan analysis functions that utilized themedical scan in their training set. The training set data 490 can alsoindicate which portions of the medical scan entry were utilized by thetraining set, for example, based on model parameter data 623 of thecorresponding medical scan analysis functions. For example, the trainingset data 490 can indicate that the medical scan image data 410 wasincluded in the training set utilized to train version X of the chestx-ray medical scan image analysis function, or that the natural languagetext data 448 of this medical scan was used to train version Y of thenatural language analysis function.

FIG. 5A presents an embodiment of a user profile entry 354, stored inuser database 344 or otherwise associated with a user. A user cancorrespond to a user of one or more of the subsystems such as aradiologist, doctor, medical professional, medical report labeler,administrator of one or more subsystems or databases, or other user thatuses one or more subsystems 101. A user can also correspond to a medicalentity such as a hospital, medical clinic, establishment that utilizesmedical scans, establishment that employs one or more of the medicalprofessionals described, an establishment associated with administeringone or more subsystems, or other entity. A user can also correspond to aparticular client device 120 or account that can be accessed one or moremedical professionals or other employees at the same or differentmedical entities. Each user profile entry can have a corresponding userprofile identifier 355.

A user profile entry 354 can include basic user data 510, which caninclude identifying information 511 corresponding to the user such as aname, contact information, account/login/password information,geographic location information such as geographic region data 424,and/or other basic information. Basic user data 510 can includeaffiliation data 512, which can list one or more medical entities orother establishments the user is affiliated with, for example, if theuser corresponds to a single person such as a medical professional, orif the user corresponds to a hospital in a network of hospitals. Theaffiliation data 512 can include one or more corresponding user profileidentifiers 355 and/or basic user data 510 if the correspondingaffiliated medical entity or other establishment has its own entry inthe user database. The user identifier data can include employee data513 listing one or more employees, such as medical professionals withtheir own user profile entries 354, for example, if the user correspondsto a medical entity or supervising medical professional of other medicalprofessional employees, and can list a user profile identifier 355and/or basic user data 510 for each employee. The basic user data 510can also include imaging machine data 514, which can include a list ofmachines affiliated with the user which can include machine identifiers,model information, calibration information, scan type information, orother data corresponding to each machine, for example, corresponding tothe machine data 425. The user profile entry can include client devicedata 515, which can include identifiers for one or more client devicesassociated with the user, for example, allowing subsystems 101 to senddata to a client device 120 corresponding to a selected user based onthe client device data and/or to determine a user that data was receivedby determining the client device from which the data was received.

The user profile entry can include usage data 520 which can includeidentifying information for a plurality of usages by the user inconjunction with using one or more subsystems 101. This can includeconsumption usage data 521, which can include a listing of, or aggregatedata associated with, usages of one or more subsystems by the user, forexample, where the user is utilizing the subsystem as a service. Forexample, the consumption usage data 521 can correspond to each instancewhere diagnosis data was sent to the user for medical scans provided tothe user in conjunction with the medical scan diagnosing system 108and/or the medical scan assisted review system 102. Some or all ofconsumption usage data 521 can include training usage data 522,corresponding to usage in conjunction with a certification program orother user training provided by one or more subsystems. The trainingusage data 522 can correspond to each instance where diagnosis feedbackdata was provided by user for a medical scan with known diagnosis data,but diagnosis feedback data is not utilized by a subsystem to generate,edit, and/or confirm diagnosis data 440 of the medical scan, as it isinstead utilized to train a user and/or determine performance data for auser.

Usage data 520 can include contribution usage data 523, which caninclude a listing of, or aggregate data associated with, usages of oneor more subsystems 101 by the user, for example, where the user isgenerating and/or otherwise providing data and/or feedback that can isutilized by the subsystems, for example, to generate, edit, and/orconfirm diagnosis data 440 and/or to otherwise populate, modify, orconfirm portions of the medical scan database or other subsystem data.For example, the contribution usage data 523 can correspond to diagnosisfeedback data received from user, used to generate, edit, and/or confirmdiagnosis data. The contribution usage data 523 can include interactiveinterface feature data 524 corresponding to the interactive interfacefeatures utilized with respect to the contribution.

The consumption usage data 521 and/or the contribution usage data 523can include medical scan entry 352 whose entries the user utilizedand/or contributed to, can indicate one or more specific attributes of amedical scan entry 352 that a user utilized and/or contributed to,and/or a log of the user input generated by a client device of the userin conjunction with the data usage. The contribution usage data 523 caninclude the diagnosis data that the user may have generated and/orreviewed, for example, indicated by, mapped to, and/or used to generatethe annotation author data 450 of corresponding medical scan entries352. Some usages may correspond to both consumption usage of theconsumption usage data 521 and contribution usage of the contributionusage data 523. The usage data 520 can also indicate one or moresubsystems 101 that correspond to each consumption and/or contribution.

The user profile entry can include performance score data 530. This caninclude one or more performance scores generated based on thecontribution usage data 523 and/or training usage data 522. Theperformance scores can include separate performance scores generated forevery contribution in the contribution usage data 523 and/or trainingusage data 522 and/or generated for every training consumption usagescorresponding to a training program. As used herein, a “high”performance score refers to a more favorable performance or rating thana “low” performance score.

The performance score data can include accuracy score data 531, whichcan be generated automatically by a subsystem for each contribution, forexample, based on comparing diagnosis data received from a user to datato known truth data such as medical scans with a truth flag 461, forexample, retrieved from the corresponding medical scan entry 352 and/orbased on other data corresponding to the medical scan, for example,received from an expert user that later reviewed the contribution usagedata of the user and/or generated automatically by a subsystem. Theaccuracy score data 531 can include an aggregate accuracy scoregenerated automatically by a subsystem, for example, based on theaccuracy data of multiple contributions by the user over time.

The performance data can also include efficiency score data 532generated automatically by a subsystem for each contribution based on anamount of time taken to complete a contribution, for example, from atime the request for a contribution was sent to the client device to atime that the contribution was received from the client device, based ontiming data received from the client device itself, and/or based onother factors. The efficiency score can include an aggregate efficiencyscore, which can be generated automatically by a subsystem based on theindividual efficiency scores over time and/or based on determining acontribution completion rate, for example based on determining how manycontributions were completed in a fixed time window.

Aggregate performance score data 533 can be generated automatically by asubsystem based on the aggregate efficiency and/or accuracy data. Theaggregate performance data can include categorized performance data 534,for example, corresponding to different scan types, different anatomicalregions, different subsystems, different interactive interface featuresand/or display parameters. The categorized performance data 534 can bedetermined automatically by a subsystem based on the scan type data 421and/or anatomical region data 422 of the medical scan associated witheach contribution, one or more subsystems 101 associated with eachcontribution, and/or interactive interface feature data 524 associatedwith each contribution. The aggregate performance data can also be basedon performance score data 530 of individual employees if the usercorresponds to a medical entity, for example, retrieved based on userprofile identifiers 355 included in the employee data 513. Theperformance score data can also include ranking data 535, which caninclude an overall ranking or categorized rankings, for example,generated automatically by a subsystem or the database itself based onthe aggregate performance data.

In some embodiments, aggregate data for each user can be further brokendown based on scores for distinct scan categories, for example, based onthe scan classifier data 420, for example, where a first aggregate datascore is generated for a user “A” based on scores from all knee x-rays,and a second aggregate data score is generated for user A based onscores from all chest CT scans. Aggregate data for each user can befurther based on scores for distinct diagnosis categories, where a firstaggregate data score is generated for user A based on scores from allnormal scans, and a second aggregate data score is generated for user Abased on scores from all scans that contain an abnormality. This can befurther broken down, where a first aggregate score is generated for userA based on all scores from scans that contain an abnormality of a firsttype and/or in a first anatomical location, and a second aggregate scoreis generated for A based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each user can be further based on affiliation data, where a rankingis generated for a medical professional “B” based on scores from allmedical professionals with the same affiliation data, and/or where aranking is generated for a hospital “C” based on scores for allhospitals, all hospitals in the same geographical region, etc. Aggregatedata for each user can be further based on scores for interfacefeatures, where a first aggregate data score is generated for user Abased on scores using a first interface feature, and a second aggregatedata score is generated for user A based on scores using a firstinterface feature.

The user profile entry can include qualification data 540. Thequalification data can include experience data 541 such as educationdata, professional practice data, number of years practicing, awardsreceived, etc. The qualification data 540 can also include certificationdata 542 corresponding to certifications earned based on contributionsto one or more subsystems, for example, assigned to users automaticallyby a subsystem based on the performance score data 530 and/or based on anumber of contributions in the contribution usage data 523 and/ortraining usage data 522. For example, the certifications can correspondto standard and/or recognized certifications to train medicalprofessionals and/or incentivize medical professionals to use thesystem. The qualification data 540 can include expert data 543. Theexpert data 543 can include a binary expert identifier, which can begenerated automatically by a subsystem based on experience data 541,certification data 542, and/or the performance score data 530, and canindicate whether the user is an expert user. The expert data 543 caninclude a plurality of categorized binary expert identifierscorresponding to a plurality of qualification categories correspondingto corresponding to scan types, anatomical regions, and/or theparticular subsystems. The categorized binary expert identifiers can begenerated automatically by a subsystem based on the categorizedperformance data 534 and/or the experience data 541. The categories beranked by performance score in each category to indicate particularspecialties. The expert data 543 can also include an expert ranking orcategorized expert ranking with respect to all experts in the system.

The user profile entry can include subscription data 550, which caninclude a selected one of a plurality of subscription options that theuser has subscribed to. For example, the subscription options cancorrespond to allowed usage of one or more subsystems, such as a numberof times a user can utilize a subsystem in a month, and/or to acertification program, for example paid for by a user to receivetraining to earn a subsystem certification of certification data 542.The subscription data can include subscription expiration information,and/or billing information. The subscription data can also includesubscription status data 551, which can for example indicate a number ofremaining usages of a system and/or available credit information. Forexample, the remaining number of usages can decrease and/or availablecredit can decrease in response to usages that utilize one or moresubsystems as a service, for example, indicated in the consumption usagedata 521 and/or training usage data 522. In some embodiments, theremaining number of usages can increase and/or available credit canincrease in response to usages that correspond to contributions, forexample, based on the contribution usage data 523. An increase in creditcan be variable, and can be based on a determined quality of eachcontribution, for example, based on the performance score data 530corresponding to the contribution where a higher performance scorecorresponds to a higher increase in credit, based on scan priority data427 of the medical scan where contributing to higher priority scanscorresponds to a higher increase in credit, or based on other factors.

The user profile entry 354 can include interface preference data 560.The interface preference data can include a preferred interactiveinterface feature set 561, which can include one or more interactiveinterface feature identifiers and/or one or more interactive interfaceversion identifiers of interface feature entries 358 and/or versionidentifiers of the interface features. Some or all of the interfacefeatures of the preferred interactive interface feature set 561 cancorrespond to display parameter data 470 of medical scans. The preferredinteractive interface feature set 561 can include a single interactivefeature identifier for one or more feature types and/or interface types,and/or can include a single interactive interface version identifier forone or more interface categories. The preferred interactive interfacefeature set 561 can include a ranking of multiple features for the samefeature type and/or interface type. The ranked and/or unranked preferredinteractive interface feature set 561 can be generated based on userinput to an interactive interface of the client device to select and/orrank some or all of the interface features and/or versions. Some or allof the features and/or versions of the preferred interactive feature setcan be selected and/or ranked automatically by a subsystem such as themedical scan interface evaluator system, for example based on interfacefeature performance score data and/or feature popularity data.Alternatively or in addition, the performance score data 530 can beutilized by a subsystem to automatically determine the preferredinteractive feature set, for example, based on the scores in differentfeature-based categories of the categorized performance data 534.

The user profile entry 354 can include report formatting data 570, whichcan indicate report formatting preferences indicated by the user. Thiscan include font, text size, header data, bulleting or numbering type,margins, file type, preferences for including one or more full orcropped image slices 412, preferences for including similar medicalscans, preferences for including additional medical scans in reports, orother formatting preference to list natural language text data and/orimage data corresponding to each abnormality. Some or all of the reportformatting data 570 can be based on interface preference data 560. Thereport formatting data 570 can be used by one or more subsystems toautomatically generate report data 449 of medical scans based on thepreferences of the requesting user.

FIG. 5B presents an embodiment of a medical scan analysis function entry356, stored in medical scan analysis function database 346 or otherwiseassociated with one of a plurality of medical scan analysis functionstrained by and/or utilized by one or more subsystems 101. For example, amedical scan analysis function can include one or more medical scanimage analysis functions trained by the medical scan image analysissystem 112; one or more medical scan natural language analysis functionstrained by the medical scan natural language analysis system 114; one ormore medical scan similarity analysis function trained by the medicalscan image analysis system 112, the medical scan natural languageanalysis system 114, and/or the medical scan comparison system 116; oneor more medical report generator functions trained by the medical scannatural language analysis system 114 and/or the medical scan imageanalysis system 112, and/or the medical report analysis function trainedby the medical scan natural language analysis system 114. Some or all ofthe medical scan analysis functions can correspond to medical scaninference functions of the medical scan diagnosing system 108, thede-identification function and/or the inference functions utilized by amedical picture archive integration system as discussed in conjunctionwith FIGS. 8A-8F, or other functions and/or processes described hereinin conjunction with one or more subsystems 101. Each medical scananalysis function entry 356 can include a medical scan analysis functionidentifier 357.

A medical scan analysis function entry 356 can include functionclassifier data 610. Function classifier data 610 can include input andoutput types corresponding to the function. For example the functionclassifier data can include input scan category 611 that indicates whichtypes of scans can be used as input to the medical scan analysisfunction. For example, input scan category 611 can indicate that amedical scan analysis function is for chest CT scans from a particularhospital or other medical entity. The input scan category 611 caninclude one or more categories included in scan classifier data 420. Invarious embodiments, the input scan category 611 corresponds to thetypes of medical scans that were used to train the medical scan analysisfunction. Function classifier data 610 can also include output type data612 that characterizes the type of output that will be produced by thefunction, for example, indicating that a medical scan analysis functionis used to generate medical codes 447. The input scan category 611 canalso include information identifying which subsystems 101 areresponsible for running the medical scan analysis function.

A medical scan analysis function entry 356 can include trainingparameters 620. This can include training set data 621, which caninclude identifiers for the data used to train the medical scan analysisfunction, such as a set of medical scan identifiers 353 corresponding tothe medical scans used to train the medical scan analysis function, alist of medical scan reports and corresponding medical codes used totrain the medical scan analysis function, etc. Alternatively or inaddition to identifying particular scans of the training set, thetraining set data 621 can identify training set criteria, such asnecessary scan classifier data 420, necessary abnormality locations,classifiers, or other criteria corresponding to abnormality annotationdata 442, necessary confidence score data 460, for example, indicatingthat only medical scans with diagnosis data 440 assigned a truth flag461 or with confidence score data 460 otherwise comparing favorably to atraining set confidence score threshold are included, a number ofmedical scans to be included and proportion data corresponding todifferent criteria, or other criteria used to populate a training setwith data of medical scans. Training parameters 620 can include modeltype data 622 indicating one or more types of model, methods, and/ortraining functions used to determine the medical scan analysis functionby utilizing the training set 621. Training parameters 620 can includemodel parameter data 623 that can include a set of features of thetraining data selected to train the medical scan analysis function,determined values for weights corresponding to selected input and outputfeatures, determined values for model parameters corresponding to themodel itself, etc. The training parameter data can also include testingdata 624, which can identify a test set of medical scans or other dataused to test the medical scan analysis function. The test set can be asubset of training set 621, include completely separate data thantraining set 621, and/or overlap with training set 621. Alternatively orin addition, testing data 624 can include validation parameters such asa percentage of data that will be randomly or pseudo-randomly selectedfrom the training set for testing, parameters characterizing a crossvalidation process, or other information regarding testing. Trainingparameters 620 can also include training error data 625 that indicates atraining error associated with the medical scan analysis function, forexample, based on applying cross validation indicated in testing data624.

A medical scan analysis function entry 356 can include performance scoredata 630. Performance data can include model accuracy data 631, forexample, generated and/or updated based on the accuracy of the functionwhen performed on new data. For example, the model accuracy data 631 caninclude or be calculated based on the model error for determined forindividual uses, for example, generated by comparing the output of themedical scan analysis function to corresponding data generated by userinput to interactive interface 275 in conjunction with a subsystem 101and/or generated by comparing the output of the medical scan analysisfunction to medical scans with a truth flag 461. The model accuracy data631 can include aggregate model accuracy data computed based on modelerror of individual uses of the function over time. The performancescore data 630 can also include model efficiency data 632, which can begenerated based on how quickly the medical scan analysis functionperforms, how much memory is utilized by medical scan analysis function,or other efficiency data relating to the medical scan analysis function.Some or all of the performance score data 630 can be based on trainingerror data 625 or other accuracy and/or efficiency data determinedduring training and/or validation. As used herein, a “high” performancescore refers to a more favorable performance or rating than a “low”performance score.

A medical scan analysis function entry 356 can include version data 640.The version data can include a version identifier 641. The version datacan indicate one or more previous version identifiers 642, which can mapto version identifiers 641 stored in other medical scan analysisfunction entry 356 that correspond to previous versions of the function.Alternatively or in addition, the version data can indicate multipleversions of the same type based on function classifier data 610, canindicate the corresponding order and/or rank of the versions, and/or canindicate training parameters 620 associated with each version.

A medical scan analysis function entry 356 can include remediation data650. Remediation data 650 can include remediation instruction data 651which can indicate the steps in a remediation process indicating how amedical scan analysis function is taken out of commission and/orreverted to a previous version in the case that remediation isnecessary. The version data 640 can further include remediation criteriadata 652, which can include threshold data or other criteria used toautomatically determine when remediation is necessary. For example, theremediation criteria data 652 can indicate that remediation is necessaryat any time where the model accuracy data and/or the model efficiencydata compares unfavorably to an indicated model accuracy thresholdand/or indicated model efficiency threshold. The remediation data 650can also include recommissioning instruction data 653, identifyingrequired criteria for recommissioning a medical scan analysis functionand/or updating a medical scan analysis function. The remediation data650 can also include remediation history, indicating one or moreinstances that the medical scan analysis function was taken out ofcommission and/or was recommissioned.

FIGS. 6A and 6B present an embodiment of a medical scan diagnosingsystem 108.

The medical scan diagnosing system 108 can generate inference data 1110for medical scans by utilizing a set of medical scan inference functions1105, stored and run locally, stored and run by another subsystem 101,and/or stored in the medical scan analysis function database 346, wherethe function and/or parameters of the function can be retrieved from thedatabase by the medical scan diagnosing system. For example, the set ofmedical scan inference function 1105 can include some or all medicalscan analysis functions described herein or other functions thatgenerate inference data 1110 based on some or all data corresponding toa medical scan such as some or all data of a medical scan entry 352.Each medical scan inference function 1105 in the set can correspond to ascan category 1120, and can be trained on a set of medical scans thatcompare favorably to the scan category 1120. For example, each inferencefunction can be trained on a set of medical scans of the one or moresame scan classifier data 420, such as the same and/or similar scantypes, same and/or similar anatomical regions locations, same and/orsimilar machine models, same and/or similar machine calibration, sameand/or similar contrasting agent used, same and/or similar originatingentity, same and/or similar geographical region, and/or otherclassifiers. Thus, the scan categories 1120 can correspond to one ormore of a scan type, scan anatomical region data, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data 420. For example, a first medical scan inferencefunction can be directed to characterizing knee x-rays, and a secondmedical scan inference function can be directed to chest CT scans. Asanother example, a first medical scan inference function can be directedto characterizing CT scans from a first hospital, and a second medicalscan image analysis function can be directed to characterizing CT scansfrom a second hospital.

Training on these categorized sets separately can ensure each medicalscan inference function 1105 is calibrated according to its scancategory 1120, for example, allowing different inference functions to becalibrated on type specific, anatomical region specific, hospitalspecific, machine model specific, and/or region-specific tendenciesand/or discrepancies. Some or all of the medical scan inferencefunctions 1105 can be trained by the medical scan image analysis systemand/or the medical scan natural language processing system, and/or somemedical scan inference functions 1105 can utilize both image analysisand natural language analysis techniques to generate inference data1110. For example, some or all of the inference functions can utilizeimage analysis of the medical scan image data 410 and/or naturallanguage data extracted from abnormality annotation data 442 and/orreport data 449 as input, and generate diagnosis data 440 such asmedical codes 447 as output. Each medical scan inference function canutilize the same or different learning models to train on the same ordifferent features of the medical scan data, with the same or differentmodel parameters, for example indicated in the model type data 622 andmodel parameter data 623. Model type and/or parameters can be selectedfor a particular medical scan inference function based on particularcharacteristics of the one or more corresponding scan categories 1120,and some or all of the indicated in the model type data 622 and modelparameter data 623 can be selected automatically by a subsystem duringthe training process based on the particular learned and/or otherwisedetermined characteristics of the one or more corresponding scancategories 1120.

As shown in FIG. 6A, the medical scan diagnosing system 108 canautomatically select a medical scan for processing in response toreceiving it from a medical entity via the network. Alternatively, themedical scan diagnosing system 108 can automatically retrieve a medicalscan from the medical scan database that is selected based on a requestreceived from a user for a particular scan and/or based on a queue ofscans automatically ordered by the medical scan diagnosing system 108 oranother subsystem based on scan priority data 427.

Once a medical scan to be processed is determined, the medical scandiagnosing system 108 can automatically select an inference function1105 based on a determined scan category 1120 of the selected medicalscan and based on corresponding inference function scan categories. Thescan category 1120 of a scan can be determined based one some or all ofthe scan classifier data 420 and/or based on other metadata associatedwith the scan. This can include determining which one of the pluralityof medical scan inference functions 1105 matches or otherwise comparesfavorably to the scan category 1120, for example, by comparing the scancategory 1120 to the input scan category of the function classifier data610.

Alternatively or in addition, the medical scan diagnosing system 108 canautomatically determine which medical scan inference function 1105 isutilized based on an output preference that corresponding to a desiredtype of inference data 1110 that is outputted by an inference function1105. The output preference designated by a user of the medical scandiagnosing system 108 and/or based on the function of a subsystem 101utilizing the medical scan diagnosing system 108. For example, the setof inference functions 1105 can include inference functions that areutilized to indicate whether or not a medical scan is normal, toautomatically identify at least one abnormality in the scan, toautomatically characterize the at least one abnormality in the scan, toassign one or more medical codes to the scan, to generate naturallanguage text data and/or a formatted report for the scan, and/or toautomatically generate other diagnosis data such as some or all ofdiagnosis data 440 based on the medical scan. Alternatively or inaddition, some inference functions can also be utilized to automaticallygenerate confidence score data 460, display parameter data 470, and/orsimilar scan data 480. The medical scan diagnosing system 108 cancompare the output preference to the output type data 612 of the medicalscan inference function 1105 to determine the selected inferencefunction 1105. For example, this can be used to decide between a firstmedical scan inference function that automatically generates medicalcodes and a second medical scan inference function that automaticallygenerates natural language text for medical reports based on the desiredtype of inference data 1110.

Prior to performing the selected medical scan inference function 1105,the medical scan diagnosing system 108 can automatically perform aninput quality assurance function 1106 to ensure the scan classifier data420 or other metadata of the medical scan accurately classifies themedical scan such that the appropriate medical scan inference function1105 of the appropriate scan category 1120 is selected. The inputquality assurance function can be trained on, for example, medical scanimage data 410 of plurality of previous medical scans with verified scancategories. Thus, the input quality assurance function 1106 can takemedical scan image data 410 as input and can generate an inferred scancategory as output. The inferred scan category can be compared to thescan category 1120 of the scan, and the input quality assurance function1106 can determine whether or not the scan category 1120 is appropriateby determining whether the scan category 1120 compares favorably to theautomatically generated inferred scan category. The input qualityassurance function 1106 can also be utilized to reassign the generatedinferred scan category to the scan category 1120 when the scan category1120 compares favorably to the automatically generated inferred scancategory. The input quality assurance function 1106 can also be utilizedto assign the generated inferred scan category to the scan category 1120for incoming medical scans that do not include any classifying data,and/or to add classifiers in scan classifier data 420 to medical scansmissing one or more classifiers.

In various embodiments, upon utilizing the input quality assurancefunction 1106 to determine that the scan category 1120 determined by ascan classifier data 420 or other metadata is inaccurate, the medicalscan diagnosing system 108 can transmit an alert and/or an automaticallygenerated inferred scan category to the medical entity indicating thatthe scan is incorrectly classified in the scan classifier data 420 orother metadata. In some embodiments, the medical scan diagnosing system108 can automatically update performance score data corresponding to theoriginating entity of the scan indicated in originating entity data 423,or another user or entity responsible for classifying the scan, forexample, where a lower performance score is generated in response todetermining that the scan was incorrectly classified and/or where ahigher performance score is generated in response to determining thatthe scan was correctly classified.

In some embodiments, the medical scan diagnosing system 108 can transmitthe medical scan and/or the automatically generated inferred scancategory to a selected user. The user can be presented the medical scanimage data 410 and/or other data of the medical scan via the interactiveinterface 275, for example, displayed in conjunction with the medicalscan assisted review system 102. The interface can prompt the user toindicate the appropriate scan category 1120 and/or prompt the user toconfirm and/or edit the inferred scan category, also presented to theuser. For example, scan review data can be automatically generated toreflect the user generated and/or verified scan category 1120, This userindicated scan category 1120 can be utilized to select to the medicalscan inference function 1105 and/or to update the scan classifier data420 or other metadata accordingly. In some embodiments, for example,where the scan review data indicates that the selected user disagreeswith the automatically generated inferred scan category created by theinput quality assurance function 1106, the medical scan diagnosingsystem 108 can automatically update performance score data 630 of theinput quality assurance function 1106 by generating a low performancescore and/or determine to enter the remediation step 1140 for the inputquality assurance function 1106.

The medical scan diagnosing system 108 can also automatically perform anoutput quality assurance step after a medical scan inference function1105 has been performed on a medical scan to produce the inference data1110, as illustrated in the embodiment presented in FIG. 6B. The outputquality assurance step can be utilized to ensure that the selectedmedical scan inference function 1105 generated appropriate inferencedata 1110 based on expert feedback. The inference data 1110 generated byperforming the selected medical scan inference function 1105 can be sentto a client device 120 of a selected expert user, such as an expert userin the user database selected based on categorized performance dataand/or qualification data that corresponds to the scan category 1120and/or the inference itself, for example, by selecting an expert userbest suited to review an identified abnormality classifier category 444and/or abnormality pattern category 446 in the inference data 1110 basedon categorized performance data and/or qualification data of acorresponding user entry. The selected user can also correspond to amedical professional or other user employed at the originating entityand/or corresponding to the originating medical professional, indicatedin the originating entity data 423.

FIG. 6B illustrates an embodiment of the medical scan diagnosing system108 in conjunction with performing a remediation step 1140. The medicalscan diagnosing system 108 can monitor the performance of the set ofmedical scan inference functions 1105, for example, based on evaluatinginference accuracy data outputted by an inference data evaluationfunction and/or based monitoring on the performance score data 630 inthe medical scan analysis function database, and can determine whetheror not if the corresponding medical scan inference function 1105 isperforming properly. This can include, for example, determining if aremediation step 1140 is necessary for a medical scan inference function1105, for example, by comparing the performance score data 630 and/orinference accuracy data to remediation criteria data 652. Determining ifa remediation step 1140 is necessary can also be based on receiving anindication from the expert user or another user that remediation isnecessary for one or more identified medical scan inference functions1105 and/or for all of the medical scan inference functions 1105.

In various embodiments, a remediation evaluation function is utilized todetermine if a remediation step 1140 is necessary for medical scaninference function 1105. The remediation evaluation function can includedetermining that remediation is necessary when recent accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below the normal performance level of the particular inferencefunction. The remediation evaluation function can include determiningthat remediation is necessary when recent or overall accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below a recent or overall average for all or similar medicalscan inference functions 1105. The remediation evaluation function caninclude determining that remediation is necessary only after a thresholdnumber of incorrect diagnoses are made. In various embodiments, multiplethreshold number of incorrect diagnoses correspond to differentdiagnoses categories. For example, the threshold number of incorrectdiagnoses for remediation can be higher for false negative diagnosesthan false positive diagnoses. Similarly, categories corresponding todifferent diagnosis severities and/or rarities can have differentthresholds, for example where a threshold number of more severe and/ormore rare diagnoses that were inaccurate to necessitate remediation islower than a threshold number of less severe and/or less rare diagnosesthat were inaccurate.

The remediation step 1140 can include automatically updating anidentified medical inference function 1105. This can includeautomatically retraining identified medical inference function 1105 onthe same training set or on a new training set that includes new data,data with higher corresponding confidence scores, or data selected basedon new training set criteria. The identified medical inference function1105 can also be updated and/or changed based on the review datareceived from the client device. For example, the medical scan andexpert feedback data can be added to the training set of the medicalscan inference function 1105, and the medical scan inference function1105 can be retrained on the updated training set. Alternatively or inaddition, the expert user can identify additional parameters and/orrules in the expert feedback data based on the errors made by theinference function in generating the inference data 1110 for the medicalscan, and these parameters and/or rules can be applied to update themedical scan inference function, for example, by updating the model typedata 622 and/or model parameter data 623.

The remediation step 1140 can also include determining to split a scancategory 1120 into two or more subcategories. Thus, two or more newmedical scan inference functions 1105 can be created, where each newmedical scan inference functions 1105 is trained on a correspondingtraining set that is a subset of the original training set and/orincludes new medical scan data corresponding to the subcategory. Thiscan allow medical scan inference functions 1105 to become morespecialized and/or allow functions to utilize characteristics and/ordiscrepancies specific to the subcategory when generating inference data1110. Similarly, a new scan category 1120 that was not previouslyrepresented by any of the medical scan inference functions 1105 can beadded in the remediation step, and a new medical scan inferencefunctions 1105 can be trained on a new set of medical scan data thatcorresponds to the new scan category 1120. Splitting a scan categoryand/or adding a scan category can be determined automatically by themedical scan diagnosing system 108 when performing the remediation step1140, for example, based on performance score data 630. This can also bedetermined based on receiving instructions to split a category and/oradd a new scan category from the expert user or other user of thesystem.

After a medical scan inference function 1105 is updated or created forthe first time, the remediation step 1140 can further undergo acommissioning test, which can include rigorous testing of the medicalscan inference function 1105 on a testing set, for example, based on thetraining parameters 620. For example, the commissioning test can bepassed when the medical scan inference function 1105 generates athreshold number of correct inference data 1110 and/or the test can bepassed if an overall or average discrepancy level between the inferencedata and the test data is below a set error threshold. The commissioningtest can also evaluate efficiency, where the medical scan inferencefunction 1105 only passes the commissioning test if it performs at orexceeds a threshold efficiency level. If the medical scan inferencefunction 1105 fails the commissioning test, the model type and/or modelparameters can be modified automatically or based on user input, and themedical scan inference function can be retested, continuing this processuntil the medical scan inference function 1105 passes the commissioningtest.

The remediation step 1140 can include decommissioning the medical scaninference function 1105, for example, while the medical scan inferencefunction is being retrained and/or is undergoing the commissioning test.Incoming scans to the medical scan diagnosing system 108 with a scancategory 1120 corresponding to a decommissioned medical scan inferencefunction 1105 can be sent directly to review by one or more users, forexample, in conjunction with the medical scan annotator system 106.These user-reviewed medical scans and corresponding annotations can beincluded in an updated training set used to train the decommissionedmedical scan inference function 1105 as part of the remediation step1140. In some embodiments, previous versions of the plurality of medicalscan image analysis functions can be stored in memory of the medicalscan diagnosing system and/or can be determined based on the versiondata 640 of a medical scan inference function 1105. A previous versionof a medical scan inference function 1105, such as most recent versionor version with the highest performance score, can be utilized duringthe remediation step 1140 as an alternative to sending all medical scansto user review.

A medical scan inference function can also undergo the remediation step1140 automatically in response to a hardware and/or software update onprocessing, memory, and/or other computing devices where the medicalscan inference function 1105 is stored and/or performed. Differentmedical scan inference functions 1105 can be containerized on their owndevices by utilizing a micro-service architecture, so hardware and/orsoftware updates may only necessitate that one of the medical scaninference functions 1105 undergo the remediation step 1140 while theothers remain unaffected. A medical scan inference function 1105 canalso undergo the remediation step 1140 automatically in response tonormal system boot-up, and/or periodically in fixed intervals. Forexample, in response to a scheduled or automatically detected hardwareand/or software update, change, or issue, one or more medical scaninference functions 1105 affected by this hardware or software can betaken out of commission until they each pass the commissioning test.Such criteria can be indicated in the remediation criteria data 652.

The medical scan diagnosing system 108 can automatically manage usagedata, subscription data, and/or billing data for the plurality of userscorresponding to user usage of the system, for example, by utilizing,generating, and/or updating some or all of the subscription data of theuser database. Users can pay for subscriptions to the system, which caninclude different subscription levels that can correspond to differentcosts. For example, a hospital can pay a monthly cost to automaticallydiagnose up to 100 medical scans per month. The hospital can choose toupgrade their subscription or pay per-scan costs for automaticdiagnosing of additional scans received after the quota is reachedand/or the medical scan diagnosing system 108 can automatically sendmedical scans received after the quota is reached to an expert userassociated with the hospital. In various embodiments incentive programscan be used by the medical scan diagnosing system to encourage expertsto review medical scans from different medical entities. For example, anexpert can receive credit to their account and/or subscription upgradesfor every medical scan reviewed, or after a threshold number of medicalscans are reviewed. The incentive programs can include interactions by auser with other subsystems, for example, based on contributions made tomedical scan entries via interaction with other subsystems.

FIG. 7A presents an embodiment of a medical scan image analysis system112. A training set of medical scans used to train one more medical scanimage analysis functions can be received from one or more client devicesvia the network and/or can be retrieved from the medical scan database342, for example, based on training set data 621 corresponding tomedical scan image analysis functions. Training set criteria, forexample, identified in training parameters 620 of the medical scan imageanalysis function, can be utilized to automatically identify and selectmedical scans to be included in the training set from a plurality ofavailable medical scans. The training set criteria can be automaticallygenerated based on, for example, previously learned criteria, and/ortraining set criteria can be received via the network, for example, froman administrator of the medical scan image analysis system. The trainingset criteria can include a minimum training set size. The training setcriteria can include data integrity requirements for medical scans inthe training set such as requiring that the medical scan is assigned atruth flag 461, requiring that performance score data for a hospitaland/or medical professional associated with the medical scan comparesfavorably to a performance score threshold, requiring that the medicalscan has been reviewed by at least a threshold number of medicalprofessionals, requiring that the medical scan and/or a diagnosiscorresponding to a patient file of the medical scan is older than athreshold elapsed time period, or based on other criteria intended toinsure that the medical scans and associated data in the training set isreliable enough to be considered “truth” data. The training set criteriacan include longitudinal requirements such the number of requiredsubsequent medical scans for the patient, multiple required types ofadditional scans for the patient, and/or other patient filerequirements.

The training set criteria can include quota and/or proportionrequirements for one or more medical scan classification data. Forexample, the training set criteria can include meeting quota and/orproportion requirements for one or more scan types and/or human bodylocation of scans, meeting quota or proportion requirements for a numberof normal medical scans and a number of medicals scans with identifiedabnormalities, meeting quota and/or proportion requirements for a numberof medical scans with abnormalities in certain locations and/or a numberof medical scans with abnormalities that meet certain size, type, orother characteristics, meeting quota and/or proportion data for a numberof medical scans with certain diagnosis or certain corresponding medicalcodes, and/or meeting other identified quota and/or proportion datarelating to metadata, patient data, or other data associated with themedical scans.

In some embodiments, multiple training sets are created to generatecorresponding medical scan image analysis functions, for example,corresponding to some or all of the set of medical scan inferencefunctions 1105. Some or all training sets can be categorized based onsome or all of the scan classifier data 420 as described in conjunctionwith the medical scan diagnosing system 108, where medical scans areincluded in a training set based on their scan classifier data 420matching the scan category of the training set. In some embodiments, theinput quality assurance function 1106 or another input check step can beperformed on medical scans selected for each training set to confirmthat their corresponding scan classifier data 420 is correct. In someembodiments, the input quality assurance function can correspond to itsown medical scan image analysis function, trained by the medical scanimage analysis system, where the input quality assurance functionutilizes high level computer vision technology to determine a scancategory 1120 and/or to confirm the scan classifier data 420 alreadyassigned to the medical scan.

In some embodiments, the training set will be used to create a singleneural network model, or other model corresponding to model type data622 and/or model parameter data 623 of the medical scan image analysisfunction that can be trained on some or all of the medical scanclassification data described above and/or other metadata, patient data,or other data associated with the medical scans. In other embodiments, aplurality of training sets will be created to generate a plurality ofcorresponding neural network models, where the multiple training setsare divided based on some or all of the medical scan classification datadescribed above and/or other metadata, patient data, or other dataassociated with the medical scans. Each of the plurality of neuralnetwork models can be generated based on the same or different learningalgorithm that utilizes the same or different features of the medicalscans in the corresponding one of the plurality of training sets. Themedical scan classifications selected to segregate the medical scansinto multiple training sets can be received via the network, for examplebased on input to an administrator client device from an administrator.The medical scan classifications selected to segregate the medical scanscan be automatically determined by the medical scan image analysissystem, for example, where an unsupervised clustering algorithm isapplied to the original training set to determine appropriate medicalscan classifications based on the output of the unsupervised clusteringalgorithm.

In embodiments where the medical scan image analysis system is used inconjunction with the medical scan diagnosing system, each of the medicalscan image analysis functions associated with each neural network modelcan correspond to one of the plurality of neural network modelsgenerated by the medical scan image analysis system. For example, eachof the plurality of neural network models can be trained on a trainingset classified on scan type, scan human body location, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data as discussed in conjunction with the medical scandiagnosing system. In embodiments where the training set classifiers arelearned, the medical scan diagnosing system can determine which of themedical scan image analysis functions should be applied based on thelearned classifying criteria used to segregate the original trainingset.

A computer vision-based learning algorithm used to create each neuralnetwork model can include selecting a three-dimensional subregion 1310for each medical scan in the training set. This three-dimensionalsubregion 1310 can correspond to a region that is “sampled” from theentire scan that may represent a small fraction of the entire scan.Recall that a medical scan can include a plurality of orderedcross-sectional image slices. Selecting a three-dimensional subregion1310 can be accomplished by selecting a proper image slice subset 1320of the plurality of cross-sectional image slices from each of theplurality of medical scans, and by further selecting a two-dimensionalsubregion 1330 from each of the selected subset of cross-sectional imageslices of the each of the medical scans. In some embodiments, theselected image slices can include one or more non-consecutive imageslices and thus a plurality of disconnected three-dimensional subregionswill be created. In other embodiments, the selected proper subset of theplurality of image slices correspond to a set of consecutive imageslices, as to ensure that a single, connected three-dimensionalsubregion is selected. In some embodiments, entire scans of the trainingset are used to train the neural network model. In such embodiment, asused herein, the three-dimensional subregion 1310 can refer to all ofthe medical scan image data 410 of a medical scan.

In some embodiments, a density windowing step can be applied to the fullscan or the selected three-dimensional subregion. The density windowingstep can include utilizing a selected upper density value cut off and/ora selected lower density value cut off, and masking pixels with highervalues than the upper density value cut off and/or masking pixels withlower values than the lower density value cut off. The upper densityvalue cut off and/or a selected lower density value cut off can bedetermined based on based on the range and/or distribution of densityvalues included in the region that includes the abnormality, and/orbased on the range and/or distribution of density values associated withthe abnormality itself, based on user input to a subsystem, based ondisplay parameter data associated with the medical scan or associatedwith medical scans of the same type, and/or can be learned in thetraining step. In some embodiments, a non-linear density windowingfunction can be applied to alter the pixel density values, for example,to stretch or compress contrast. In some embodiments, this densitywindowing step can be performed as a data augmenting step, to createadditional training data for a medical scan in accordance with differentdensity windows.

Having determined the subregion training set 1315 of three-dimensionalsubregions 1310 corresponding to the set of full medical scans in thetraining set, the medical scan image analysis system can complete atraining step 1352 by performing a learning algorithm on the pluralityof three-dimensional subregions to generate model parameter data 1355 ofa corresponding learning model. The learning model can include one ormore of a neural network, an artificial neural network, a convolutionalneural network, a Bayesian model, a support vector machine model, acluster analysis model, or other supervised or unsupervised learningmodel. The model parameter data 1355 can generated by performing thelearning algorithm 1350, and the model parameter data 1355 can beutilized to determine the corresponding medical scan image analysisfunctions. For example, some or all of the model parameter data 1355 canbe mapped to the medical scan analysis function in the model parameterdata 623 or can otherwise define the medical scan analysis function.

The training step 1352 can include creating feature vectors for eachthree-dimensional subregion of the training set for use by the learningalgorithm 1350 to generate the model parameter data 1355. The featurevectors can include the pixel data of the three-dimensional subregionssuch as density values and/or grayscale values of each pixel based on adetermined density window. The feature vectors can also include otherfeatures as additional input features or desired output features, suchas known abnormality data such as location and/or classification data,patient history data such as risk factor data or previous medical scans,diagnosis data, responsible medical entity data, scan machinery model orcalibration data, contrast agent data, medical code data, annotationdata that can include raw or processed natural language text data, scantype and/or anatomical region data, or other data associated with theimage, such as some or all data of a medical scan entry 352. Featurescan be selected based on administrator instructions received via thenetwork and/or can be determined based on determining a feature set thatreduces error in classifying error, for example, by performing across-validation step on multiple models created using different featuresets. The feature vector can be split into an input feature vector andoutput feature vector. The input feature vector can include data thatwill be available in subsequent medical scan input, which can includefor example, the three-dimensional subregion pixel data and/or patienthistory data. The output feature vector can include data that will beinferred in in subsequent medical scan input and can include singleoutput value, such as a binary value indicating whether or not themedical scan includes an abnormality or a value corresponding to one ofa plurality of medical codes corresponding to the image. The outputfeature vector can also include multiple values which can includeabnormality location and/or classification data, diagnosis data, orother output. The output feature vector can also include a determinedupper density value cut off and/or lower density value cut off, forexample, characterizing which pixel values were relevant to detectingand/or classifying an abnormality. Features included in the outputfeature vector can be selected to include features that are known in thetraining set, but may not be known in subsequent medical scans such astriaged scans to be diagnosed by the medical scan diagnosing system,and/or scans to be labeled by the medical scan report labeling system.The set of features in the input feature vector and output featurevector, as well as the importance of different features where eachfeature is assigned a corresponding weight, can also be designated inthe model parameter data 1355.

Consider a medical scan image analysis function that utilizes a neuralnetwork. The neural network can include a plurality of layers, whereeach layer includes a plurality of neural nodes. Each node in one layercan have a connection to some or all nodes in the next layer, where eachconnection is defined by a weight value. Thus, the model parameter data1355 can include a weight vector that includes weight values for everyconnection in the network. Alternatively or in addition, the modelparameter data 1355 can include any vector or set of parametersassociated with the neural network model, which can include an upperdensity value cut off and/or lower density value cut off used to masksome of the pixel data of an incoming image, kernel values, filterparameters, bias parameters, and/or parameters characterizing one ormore of a plurality of convolution functions of the neural networkmodel. The medical scan image analysis function can be utilized toproduce the output vector as a function of the input feature vector andthe model parameter data 1355 that characterizes the neural networkmodel. In particular, the medical scan image analysis function caninclude performing a forward propagation step plurality of neuralnetwork layers to produce an inferred output vector based on the weightvector or other model parameter data 1355. Thus, the learning algorithm1350 utilized in conjunction with a neural network model can includedetermining the model parameter data 1355 corresponding to the neuralnetwork model, for example, by populating the weight vector with optimalweights that best reduce output error.

In particular, determining the model parameter data 1355 can includeutilizing a backpropagation strategy. The forward propagation algorithmcan be performed on at least one input feature vector corresponding toat least one medical scan in the training set to propagate the at leastone input feature vector through the plurality of neural network layersbased on initial and/or default model parameter data 1355, such as aninitial weight vector of initial weight values set by an administratoror chosen at random. The at least one output vector generated byperforming the forward propagation algorithm on the at least one inputfeature vector can be compared to the corresponding at least one knownoutput feature vector to determine an output error. Determining theoutput error can include, for example, computing a vector distance suchas the Euclidian distance, or squared Euclidian distance, between theproduced output vector and the known output vector, and/or determiningan average output error such as an average Euclidian distance or squaredEuclidian distance if multiple input feature vectors were employed.Next, gradient descent can be performed to determine an updated weightvector based on the output error or average output error. This gradientdescent step can include computing partial derivatives for the errorwith respect to each weight, or other parameter in the model parameterdata 1355, at each layer starting with the output layer. Chain rule canbe utilized to iteratively compute the gradient with respect to eachweight or parameter at each previous layer until all weight's gradientsare computed. Next updated weights, or other parameters in the modelparameter data 1355, are generated by updating each weight based on itscorresponding calculated gradient. This process can be repeated on atleast one input feature vector, which can include the same or differentat least one feature vector used in the previous iteration, based on theupdated weight vector and/or other updated parameters in the modelparameter data 1355 to create a new updated weight vector and/or othernew updated parameters in the model parameter data 1355. This processcan continue to repeat until the output error converges, the outputerror is within a certain error threshold, or another criterion isreached to determine the most recently updated weight vector and/orother model parameter data 1355 is optimal or otherwise determined forselection.

Having determined the medical scan neural network and its final othermodel parameter data 1355, an inference step 1354 can be performed onnew medical scans to produce inference data 1370, such as inferredoutput vectors, as shown in FIG. 7B. The inference step can includeperforming the forward propagation algorithm to propagate an inputfeature vector through a plurality of neural network layers based on thefinal model parameter data 1355, such as the weight values of the finalweight vector, to produce the inference data. This inference step 1354can correspond to performing the medical scan image analysis function,as defined by the final model parameter data 1355, on new medical scansto generate the inference data 1370, for example, in conjunction withthe medical scan diagnosing system 108 to generate inferred diagnosisdata or other selected output data for triaged medical scans based onits corresponding the input feature vector.

The inference step 1354 can include applying the density windowing stepto new medical scans. Density window cut off values and/or a non-lineardensity windowing function that are learned can be automatically appliedwhen performing the inference step. For example, if the training step1352 was used to determine optimal upper density value cut off and/orlower density value cut off values to designate an optimal densitywindow, the inference step 1354 can include masking pixels of incomingscans that fall outside of this determined density window beforeapplying the forward propagation algorithm. As another example, iflearned parameters of one or more convolutional functions correspond tothe optimal upper density value cut off and/or lower density value cutoff values, the density windowing step is inherently applied when theforward propagation algorithm is performed on the new medical scans.

In some embodiments where a medical scan analysis function is defined bymodel parameter data 1355 corresponding to a neutral network model, theneural network model can be a fully convolutional neural network. Insuch embodiments, only convolution functions are performed to propagatethe input feature vector through the layers of the neural network in theforward propagation algorithm. This enables the medical scan imageanalysis functions to process input feature vectors of any size. Forexample, as discussed herein, the pixel data corresponding to thethree-dimensional subregions is utilized input to the forwardpropagation algorithm when the training step 1352 is employed topopulate the weight vector and/or other model parameter data 1355.However, when performing the forward propagation algorithm in theinference step 1354, the pixel data of full medical scans can beutilized as input, allowing the entire scan to be processed to detectand/or classify abnormalities, or otherwise generate the inference data1370. This may be a preferred embodiment over other embodiments wherenew scans must also be sampled by selecting a three-dimensionalsubregions and/or other embodiments where the inference step requires“piecing together” inference data 1370 corresponding to multiplethree-dimensional subregions processed separately.

The inferred output vector of the inference data 1370 can include aplurality of abnormality probabilities mapped to a pixel location ofeach of a plurality of cross-sectional image slices of the new medicalscan. For example, the inferred output vector can indicate a set ofprobability matrices 1371, where each matrix in the set corresponds toone of the plurality of image slices of the medical scan, where eachmatrix is a size corresponding to the number of pixels in each imageslice, where each cell of each matrix corresponds to a pixel of thecorresponding image slice, whose value is the abnormality probability ofthe corresponding pixel.

A detection step 1372 can include determining if an abnormality ispresent in the medical scan based on the plurality of abnormalityprobabilities. Determining if an abnormality is present can include, forexample, determining that a cluster of pixels in the same region of themedical scan correspond to high abnormality probabilities, for example,where a threshold proportion of abnormality probabilities must meet orexceed a threshold abnormality probability, where an average abnormalityprobability of pixels in the region must meet or exceed a thresholdabnormality probability, where the region that includes the cluster ofpixels must be at least a certain size, etc. Determining if anabnormality is present can also include calculating a confidence scorebased on the abnormality probabilities and/or other data correspondingto the medical scan such as patient history data. The location of thedetected abnormality can be determined in the detection step 1372 basedon the location of the pixels with the high abnormality probabilities.The detection step can further include determining an abnormality region1373, such as a two-dimensional subregion on one or more image slicesthat includes some or all of the abnormality. The abnormality region1373 determined in the detection step 1372 can be mapped to the medicalscan to populate some or all of the abnormality location data 443 foruse by one or more other subsystems 101 and/or client devices 120.Furthermore, determining whether or not an abnormality exists in thedetection step 1372 can be used to populate some or all of the diagnosisdata 440 of the medical scan, for example, to indicate that the scan isnormal or contains an abnormality in the diagnosis data 440.

An abnormality classification step 1374 can be performed on a medicalscan in response to determining an abnormality is present.Classification data 1375 corresponding to one or more classificationcategories such as abnormality size, volume, pre-post contract, doublingtime, calcification, components, smoothness, texture, diagnosis data,one or more medical codes, a malignancy rating such as a Lung-RADSscore, or other classifying data as described herein can be determinedbased on the detected abnormality. The classification data 1375generated by the abnormality classification step 1374 can be mapped tothe medical scan to populate some or all of the abnormalityclassification data 445 of the corresponding abnormality classifiercategories 444 and/or abnormality pattern categories 446 and/or todetermine one or more medical codes 447 of the medical scan. Theabnormality classification step 1374 can include performing anabnormality classification function on the full medical scan, or theabnormality region 1373 determined in the detection step 1372. Theabnormality classification function can be based on another modeltrained on abnormality data such as a support vector machine model,another neural network model, or any supervised classification modeltrained on medical scans, or portions of medical scans, that includeknown abnormality classifying data to generate inference data for someor all of the classification categories. For example, the abnormalityclassification function can include another medical scan analysisfunction. Classification data 1375 in each of a plurality ofclassification categories can also be assigned their own calculatedconfidence score, which can also be generated by utilizing theabnormality classification function. Output to the abnormalityclassification function can also include at least one identified similarmedical scan and/or at least one identified similar cropped image, forexample, based on the training data. The abnormality classification stepcan also be included in the inference step 1354, where the inferredoutput vector or other inference data 1370 of the medical scan imageanalysis function includes the classification data 1375.

The abnormality classification function can be trained on full medicalscans and/or one or more cropped or full selected image slices frommedical scans that contain an abnormality. For example, the abnormalityclassification function can be trained on a set of two-dimensionalcropped slices that include abnormalities. The selected image slicesand/or the cropped region in each selected image slice for each scan inthe training set can be automatically selected based upon the knownlocation of the abnormality. Input to the abnormality classificationfunction can include the full medical scan, one or more selected fullimage slices, and/or one or more selected image slices cropped based ona selected region. Thus, the abnormality classification step can includeautomatically selecting one or more image slices that include thedetected abnormality. The slice selection can include selecting thecenter slice in a set of consecutive slices that are determined toinclude the abnormality or selecting a slice that has the largestcross-section of the abnormality, or selecting one or more slices basedon other criteria. The abnormality classification step can also includeautomatically generating one or more cropped two-dimensional imagescorresponding to the one or more of the selected image slices based onan automatically selected region that includes the abnormality.

Input to the abnormality classification function can also include otherdata associated with the medical scan, including patient history, riskfactors, or other metadata. The abnormality classification step can alsoinclude determining some or all of the characteristics based on data ofthe medical scan itself. For example, the abnormality size and volumecan be determined based on a number of pixels determined to be part ofthe detected abnormality. Other classifiers such as abnormality textureand/or smoothness can be determined by performing one or more otherpreprocessing functions on the image specifically designed tocharacterize such features. Such preprocessed characteristics can beincluded in the input to the abnormality classification function to themore difficult task of assigning a medical code or generating otherdiagnosis data. The training data can also be preprocessed to includesuch preprocessed features.

A similar scan identification step 1376 can also be performed on amedical scan with a detected abnormality and/or can be performed on theabnormality region 1373 determined in the detection step 1372. Thesimilar scan identification step 1376 can include generating similarabnormality data 1377, for example, by identifying one or more similarmedical scans or one or more similar cropped two-dimensional images froma database of medical scans and/or database of cropped two-dimensionalimages. Similar medical scans and/or cropped images can include medicalscans or cropped images that are visually similar, medical scans orcropped images that have known abnormalities in a similar location to aninferred abnormality location of the given medical scan, medical scansthat have known abnormalities with similar characteristics to inferredcharacteristics of an abnormality in the given scan, medical scans withsimilar patient history and/or similar risk factors, or some combinationof these factors and/or other known and/or inferred factors. The similarabnormality data 1377 can be mapped to the medical scan to populate someor all of its corresponding similar scan data 480 for use by one or moreother subsystems 101 and/or client devices 120.

The similar scans identification step 1376 can include performing a scansimilarity algorithm, which can include generating a feature vector forthe given medical scan and for medical scans in the set of medicalscans, where the feature vector can be generated based on quantitativeand/or category based visual features, inferred features, abnormalitylocation and/or characteristics such as the predetermined size and/orvolume, patient history and/or risk factor features, or other known orinferred features. A medical scan similarity analysis function can beapplied to the feature vector of the given medical scan and one or morefeature vectors of medical scans in the set. The medical scan similarityanalysis function can include computing a similarity distance such asthe Euclidian distance between the feature vectors, and assigning thesimilarity distance to the corresponding medical scan in the set.Similar medical scans can be identified based on determining one or moremedical scans in the set with a smallest computed similarity distance,based on ranking medical scans in the set based on the computedsimilarity distances and identifying a designated number of top rankedmedical scans, and/or based on determining if a similarity distancebetween the given medical scan and a medical scan in the set is smallerthan a similarity threshold. Similar medical scans can also beidentified based on determining medical scans in a database that mappedto a medical code that matches the medical code of the medical scan, ormapped to other matching classifying data. A set of identified similarmedical scans can also be filtered based on other inputted orautomatically generated criteria, where for example only medical scanswith reliable diagnosis data or rich patient reports, medical scans withcorresponding with longitudinal data in the patient file such asmultiple subsequent scans taken at later dates, medical scans withpatient data that corresponds to risk factors of the given patient, orother identified criteria, where only a subset of scans that comparefavorably to the criteria are selected from the set and/or only ahighest ranked single scan or subset of scans are selected from the set,where the ranking is automatically computed based on the criteria.Filtering the similar scans in this fashion can include calculating, orcan be based on previously calculated, one or more scores as discussedherein. For example, the ranking can be based on a longitudinal qualityscore, such as the longitudinal quality score 434, which can becalculated for an identified medical scan based on a number ofsubsequent and/or previous scans for the patient. Alternatively or inaddition, the ranking can be based on a confidence score associated withdiagnosis data of the scan, such as confidence score data 460, based onperformance score data associated with a user or medical entityassociated with the scan, based on an amount of patient history data ordata in the medical scan entry 352, or other quality factors. Theidentified similar medical scans can be filtered based on ranking thescans based on their quality score and/or based on comparing theirquality score to a quality score threshold. In some embodiments, alongitudinal threshold must be reached, and only scans that comparefavorably to the longitudinal threshold will be selected. For example,only scans with at least three scans on file for the patient and finalbiopsy data will be included.

In some embodiments, the similarity algorithm can be utilized inaddition to or instead of the trained abnormality classificationfunction to determine some or all of the inferred classification data1375 of the medical scan, based on the classification data such asabnormality classification data 445 or other diagnosis data 440 mappedto one or more of the identified similar scans. In other embodiments,the similarity algorithm is merely used to identify similar scans forreview by medical professionals to aid in review, diagnosis, and/orgenerating medical reports for the medical image.

A display parameter step 1378 can be performed based on the detectionand/or classification of the abnormality. The display parameter step caninclude generating display parameter data 1379, which can includeparameters that can be used by an interactive interface to best displayeach abnormality. The same or different display parameters can begenerated for each abnormality. The display parameter data generated inthe display parameter step 1378 can be mapped to the medical scan topopulate some or all of its corresponding display parameter data 470 foruse by one or more other subsystems 101 and/or client devices 120.

Performing the display parameter step 1378 can include selecting one ormore image slices that include the abnormality by determining the one ormore image slices that include the abnormality and/or determining one ormore image slices that has a most optimal two-dimensional view of theabnormality, for example by selecting the center slice in a set ofconsecutive slices that are determined to include the abnormality,selecting a slice that has the largest cross-section of the abnormality,selecting a slice that includes a two-dimensional image of theabnormality that is most similar to a selected most similartwo-dimensional-image, selecting the slice that was used as input to theabnormality classification step and/or similar scan identification step,or based on other criteria. This can also include automatically croppingone or more selected image slices based on an identified region thatincludes the abnormality. This can also select an ideal Hounsfieldwindow that best displays the abnormality. This can also includeselecting other display parameters based on data generated by themedical scan interface evaluating system and based on the medical scan.

FIGS. 8A-8F illustrate embodiments of a medical picture archiveintegration system 2600. The medical picture archive integration system2600 can provide integration support for a medical picture archivesystem 2620, such as a PACS that stores medical scans. The medicalpicture archive integration system 2600 can utilize model parametersreceived from a central server system 2640 via a network 2630 to performan inference function on de-identified medical scans of medical scansreceived from the medical picture archive system 2620. The annotationdata produced by performing the inference function can be transmittedback to the medical picture archive system. Furthermore, the annotationdata and/or de-identified medical scans can be sent to the centralserver system 2640, and the central server system can train on thisinformation to produce new and/or updated model parameters fortransmission back to the medical picture archive integration system 2600for use on subsequently received medical scans.

In various embodiments, medical picture archive integration system 2600includes a de-identification system that includes a first memorydesignated for protected health information (PHI), operable to perform ade-identification function on a DICOM image, received from a medicalpicture archive system, to identify at least one patient identifier andgenerate a de-identified medical scan that does not include the at leastone patient identifier. The medical picture archive integration systemfurther includes a de-identified image storage system that stores thede-identified medical scan in a second memory that is separate from thefirst memory, and an annotating system, operable to utilize modelparameters received from a central server to perform an inferencefunction on the de-identified medical scan, retrieved from the secondmemory to generate annotation data for transmission to the medicalpicture archive system as an annotated DICOM file.

The first memory and the second memory can be implemented by utilizingseparate storage systems: the first memory can be implemented by a firststorage system designated for PHI storage, and the second memory can beimplemented by a second storage system designated for storage ofde-identified data. The first storage system can be protected fromaccess by the annotating system, while the second storage system can beaccessible by the annotating system. The medical picture archiveintegration system 2600 can be operable to perform the de-identificationfunction on data in first storage system to generate de-identified data.The de-identified data can then be stored in the second storage systemfor access by the annotating system. The first and second storagesystems can be physically separate, each utilizing at least one of theirown, separate memory devices. Alternatively, the first and secondstorage systems can be virtually separate, where data is stored inseparate virtual memory locations on the same set of memory devices.Firewalls, virtual machines, and/or other protected containerization canbe utilized to enforce the separation of data in each storage system, toprotect the first storage system from access by the annotating systemand/or from other unauthorized access, and/or to ensure that only dataof the first storage system that has been properly de-identified throughapplication of the de-identification function can be stored in thesecond storage system.

As shown in FIG. 8A, the medical picture archive system 2620 can receiveimage data from a plurality of modality machines 2622, such as CTmachines, MRI machines, x-ray machines, and/or other medical imagingmachines that produce medical scans. The medical picture archive system2620 can store this image data in a DICOM image format and/or can storethe image data in a plurality of medical scan entries 352 as describedin conjunction with some or all of the attributes described inconjunction with FIGS. 4A and 4B. While “DICOM image” will be usedherein to refer to medical scans stored by the medical picture archivesystem 2620, the medical picture archive integration system 2600 canprovide integration support for medical picture archive systems 2620that store medical scans in other formats.

The medical picture archive integration system 2600 can include areceiver 2602 and a transmitter 2604, operable to transmit and receivedata from the medical picture archive system 2620, respectively. Forexample, the receiver 2602 and transmitter 2604 can be configured toreceive and transmit data, respectively, in accordance with a DICOMcommunication protocol and/or another communication protocol recognizedby the medical image archive system 2620. The receiver can receive DICOMimages from the medical picture archive system 2620. The transmitter2604 can send annotated DICOM files to the medical picture archivesystem 2620.

DICOM images received via receiver 2602 can be sent directly to ade-identification system 2608. The de-identification system 2608 can beoperable to perform a de-identification function on the first DICOMimage to identify at least one patient identifier in the DICOM image,and to generate a de-identified medical scan that does not include theidentified at least one patient identifier. As used herein, a patientidentifier can include any patient identifying data in the image data,header, and/or metadata of a medical scan, such as a patient ID numberor other unique patient identifier, an accession number, aservice-object pair (SOP) instance unique identifier (UID) field, scandate and/or time that can be used to determine the identity of thepatient that was scanned at that date and/or time, and/or other privatedata corresponding to the patient, doctor, or hospital. In someembodiments, the de-identified medical scan is still in a DICOM imageformat. For example, a duplicate DICOM image that does not include thepatient identifiers can be generated, and/or the original DICOM imagecan be altered such that the patient identifiers of the new DICOM imageare masked, obfuscated, removed, replaced with a custom fiducial, and/orotherwise anonymized. In other embodiments, the de-identified medicalscan is formatted in accordance with a different image format and/ordifferent data format that does not include the identifying information.In some embodiments, other private information, for example, associatedwith a particular doctor or other medical professional, can beidentified and anonymized as well.

Some patient identifying information can be included in a DICOM headerof the DICOM image, for example, in designated fields for patientidentifiers. These corresponding fields can be anonymized within thecorresponding DICOM header field. Other patient identifying informationcan be included in the image itself, such as in medical scan image data410. For example, the image data can include a patient name or otheridentifier that was handwritten on a hard copy of the image before theimage was digitized. As another example, a hospital administered armbandor other visual patient information in the vicinity of the patient mayhave been captured in the image itself. A computer vision model candetect the presence of these identifiers for anonymization, for example,where a new DICOM image includes a fiducial image that covers theidentifying portion of the original DICOM image. In some embodiments,patient information identified in the DICOM header can be utilized todetect corresponding patient information in the image itself. Forexample, a patient name extracted from the DICOM header beforeanonymization can be used to search for the patient name in the imageand/or to detect a location of the image that includes the patient name.In some embodiments, the de-identification system 2608 is implemented bythe de-identification system discussed in conjunction with FIGS. 10A,10B and 11, and/or utilizes functions and/or operations discussed inconjunction with FIGS. 10A, 10B and 11.

The de-identified medical scan can be stored in de-identified imagestorage system 2610 and the annotating system 2612 can access thede-identified medical scan from the de-identified image storage system2610 for processing. The de-identified storage system can archive aplurality of de-identified DICOM images and/or can serve as temporarystorage for the de-identified medical scan until processing of thede-identified medical scan by the annotating system 2612 is complete.The annotating system 2612 can generate annotation data by performing aninference function on the de-identified medical scan, utilizing themodel parameters received from the central server system 2640. Theannotation data can correspond to some or all of the diagnosis data 440as discussed in conjunction with FIGS. 4A and 4B. In come embodiments,the annotating system 2612 can utilize the model parameters to performinference step 1354, the detection step 1372, the abnormalityclassification step 1374, the similar scan identification step 1376,and/or the display parameter step 1378 of the medical scan imageanalysis system 112, as discussed in conjunction with FIG. 7B, onde-identified medical scans received from the medical picture archivesystem 2620.

In some embodiments, model parameters for a plurality of inferencefunctions can be received from the central server system 2640, forexample, where each inference function corresponds to one of a set ofdifferent scan categories. Each scan category can correspond to a uniquecombination of one or a plurality of scan modalities, one of a pluralityof anatomical regions, and/or other scan classifier data 420. Forexample, a first inference function can be trained on and intended forde-identified medical scans corresponding chest CT scans, and a secondinference function can be trained on and intended for de-identifiedmedical scans corresponding to head MM scans. The annotating system canselect one of the set of inference functions based on determining thescan category of the DICOM image, indicated in the de-identified medicalscan, and selecting the inference function that corresponds to thedetermined scan category.

To ensure that scans received from the medical picture archive system2620 match the set of scan categories for which the annotating system isoperable to perform a corresponding inference function, the transmittercan transmit requests, such as DICOM queries, indicating image typeparameters such as parameters corresponding to scan classifier data 420,for example indicating one or more scan modalities, one or moreanatomical regions, and/or other parameters. For example, the requestcan indicate that all incoming scans that match the set of scancategories corresponding to a set of inference functions the annotatingsystem 2612 for which the annotating system has obtained modelparameters from the central server system 2640 and is operable toperform.

Once the annotation data is generated by performing the selectedinference function, the annotating system 2612 can generate an annotatedDICOM file for transmission to the medical image archive system 2620 forstorage. The annotated DICOM file can include some or all of the fieldsof the diagnosis data 440 and/or abnormality annotation data 442 ofFIGS. 4A and 4B. The annotated DICOM file can include scan overlay data,providing location data of an identified abnormality and/or display datathat can be used in conjunction with the original DICOM image toindicate the abnormality visually in the DICOM image and/or to otherwisevisually present the annotation data, for example, for use with themedical scan assisted review system 102. For example, a DICOMpresentation state file can be generated to indicate the location of anabnormality identified in the de-identified medical scan. The DICOMpresentation state file can include an identifier of the original DICOMimage, for example, in metadata of the DICOM presentation state file, tolink the annotation data to the original DICOM image. In otherembodiments, a full, duplicate DICOM image is generated that includesthe annotation data with an identifier linking this duplicate annotatedDICOM image to the original DICOM image.

The identifier linking the annotated DICOM file to the original DICOMimage can be extracted from the original DICOM file by thede-identification system 2608, thus enabling the medical picture archivesystem 2620 to link the annotated DICOM file to the original DICOM imagein its storage. For example, the de-identified medical scan can includean identifier that links the de-identified medical scan to the originalDICOM file, but does not link the de-identified medical scan to apatient identifier or other private data.

In some embodiments, generating the annotated DICOM file includesaltering one or more fields of the original DICOM header. For example,standardized header formatting function parameters can be received fromthe central server system and can be utilized by the annotating systemto alter the original DICOM header to match a standardized DICOM headerformat. The standardized header formatting function can be trained in asimilar fashion to other medical scan analysis functions discussedherein and/or can be characterized by some or all fields of a medicalscan analysis function entry 356. The annotating system can perform thestandardized header formatting function on a de-identified medical scanto generate a new, standardized DICOM header for the medical scan to besent back to the medical picture archive system 2620 in the annotatedDICOM file and/or to replace the header of the original DICOM file. Thestandardized header formatting function can be run in addition to otherinference functions utilized to generate annotation data. In otherembodiments, the medical picture archive integration system 2600 isimplemented primarily for header standardization for medical scansstored by the medical picture archive system 2620. In such embodiments,only the standardized header formatting function is performed on thede-identified data to generate a modified DICOM header for the originalDICOM image, but the de-identified medical scan is not annotated.

In some embodiments of header standardization, the annotation system canstore a set of acceptable, standardized entries for some or all of theDICOM header fields, and can select one of the set of acceptable,standardized entries in populating one or more fields of the new DICOMheader for the annotated DICOM file. For example, each of the set ofscan categories determined by the annotating system can correspond to astandardized entry of one or more fields of the DICOM header. The newDICOM header can thus be populated based on the determined scancategory.

In some embodiments, each of the set of standardized entries can bemapped to a set of related, non-standardized entries, such as entries ina different order, commonly misspelled entries, or other similar entriesthat do not follow a standardized format. For example, one of the set ofacceptable, standardized entries for a field corresponding to a scancategory can include “Chest CT”, which can be mapped to a set ofsimilar, non-standardized entries which can include “CT chest”,“computerized topography CT”, and/or other entries that are notstandardized. In such embodiments, the annotating system can determinethe original DICOM header is one of the similar non-standardizedentries, and can select the mapped, standardized entry as the entry forthe modified DICOM header. In other embodiments, the image data itselfand/or or other header data can be utilized by the annotation system todetermine a standardized field. For example, an input quality assurancefunction 1106 can be trained by the central server system and sent tothe annotating system to determine one or more appropriate scanclassifier fields, or one or more other DICOM header fields, based onthe image data or other data of the de-identified medical scan. One ormore standardized labels can be assigned to corresponding fields of themodified DICOM header based on the one or more fields determined by theinput quality assurance function.

In some embodiments, the DICOM header is modified based on theannotation data generated in performing the inference function. Inparticular, a DICOM priority header field can be generated and/ormodified automatically based on the severity and/or time-sensitivity ofthe abnormalities detected in performing the inference function. Forexample, a DICOM priority header field can be changed from a lowpriority to a high priority in response to annotation data indicating abrain bleed in the de-identified medical scan of a DICOM imagecorresponding to a head CT scan, and a new DICOM header that includesthe high priority DICOM priority header field can be sent back to themedical picture archive system 2620 to replace or otherwise be mapped tothe original DICOM image of the head CT scan.

In various embodiments, the medical picture archive system 2620 isdisconnected from network 2630, for example, to comply with requirementsregarding Protected Health Information (PHI), such as patientidentifiers and other private patient information included in the DICOMimages and/or otherwise stored by the medical picture archive system2620. The medical picture archive integration system 2600 can enableprocessing of DICOM images while still protecting private patientinformation by first de-identifying DICOM data by utilizingde-identification system 2608. The de-identification system 2608 canutilize designated processors and memory of the medical picture archiveintegration system, for example, designated for PHI. Thede-identification system 2608 can be decoupled from the network 2630 toprevent the DICOM images that still include patient identifiers frombeing accessed via the network 2630. For example, as shown in FIG. 8A,the de-identification system 2608 is not connected to network interface2606. Furthermore, only the de-identification system 2608 has access tothe original DICOM files received from the medical picture archivesystem 2620 via receiver 2602. The de-identified image storage system2610 and annotating system 2612, as they are connected to network 2630via network interface 2606, only store and have access to thede-identified medical scan produced by the de-identification system2608.

This containerization that separates the de-identification system 2608from the de-identified image storage system 2610 and the annotatingsystem 2612 is further illustrated in FIG. 8B, which presents anembodiment of the medical picture archive integration system 2600. Thede-identification system 2608 can include its own designated memory 2654and processing system 2652, connected to receiver 2602 via bus 2659. Forexample, this memory 2654 and processing system 2652 can be designatedfor PHI, and can adhere to requirements for handling PHI. The memory2654 can store executable instructions that, when executed by theprocessing system 2652, enable the de-identification system to performthe de-identification function on DICOM images received via receiver2602 of the de-identification system. The incoming DICOM images can betemporarily stored in memory 2654 for processing, and patientidentifiers detected in performing the de-identification function can betemporarily stored in memory 2654 to undergo anonymization. Interface2655 can transmit the de-identified medical scan to interface 2661 foruse by the de-identified image storage system 2610 and the annotatingsystem 2612. Interface 2655 can be protected from transmitting originalDICOM files and can be designated for transmission of de-identifiedmedical scan only.

Bus 2669 connects interface 2661, as well as transmitter 2604 andnetwork interface 2606, to the de-identified image storage system 2610and the annotating system 2612. The de-identified image storage system2610 and annotating system 2612 can utilize separate processors andmemory, or can utilize shared processors and/or memory. For example, thede-identified image storage system 2610 can serve as temporary memory ofthe annotating system 2612 as de-identified images are received andprocessed to generate annotation data.

As depicted in FIG. 8B, the de-identified image storage system 2610 caninclude memory 2674 that can temporarily store incoming de-identifiedmedical scans as it undergoes processing by the annotating system 2612and/or can archive a plurality of de-identified medical scanscorresponding to a plurality of DICOM images received by the medicalpicture archive integration system 2600. The annotating system 2612 caninclude a memory 2684 that stores executable instructions that, whenexecuted by processing system 2682, cause the annotating system 2612perform a first inference function on de-identified medical scan togenerate annotation data by utilizing the model parameters received viainterface 2606, and to generate an annotated DICOM file based on theannotation data for transmission via transmitter 2604. The modelparameters can be stored in memory 2684, and can include modelparameters for a plurality of inference functions, for example,corresponding to a set of different scan categories.

The medical picture archive integration system can be an onsite system,installed at a first geographic site, such as a hospital or othermedical entity that is affiliated with the medical picture archivesystem 2620. The hospital or other medical entity can further beresponsible for the PHI of the de-identification system, for example,where the memory 2654 and processing system 2652 are owned by,maintained by, and/or otherwise affiliated with the hospital or othermedical entity. The central server system 2640 can be located at asecond, separate geographic site that is not affiliated with thehospital or other medical entity and/or at a separate geographic sitethat is not affiliated with the medical picture archive system 2620. Thecentral server system 2640 can be a server configured to be outside thenetwork firewall and/or out outside the physical security of thehospital or other medical entity or otherwise not covered by theparticular administrative, physical and technical safeguards of thehospital or other medical entity.

FIG. 8C further illustrates how model parameters can be updated overtime to improve existing inference functions and/or to add new inferencefunctions, for example corresponding to new scan categories. Inparticular, the some or all of the de-identified medical scans generatedby the de-identification system 2608 can be transmitted back to thecentral server system, and the central server system 2640 can train onthis data to improve existing models by producing updated modelparameters of an existing inference function and/or to generate newmodels, for example, corresponding to new scan categories, by producingnew model parameters for new inference functions. For example, thecentral server system 2640 can produce updated and/or new modelparameters by performing the training step 1352 of the medical scanimage analysis system 112, as discussed in conjunction with FIG. 7A, ona plurality of de-identified medical scans received from the medicalpicture archive integration system 2600.

The image type parameters can be determined by the central server systemto dictate characteristics of the set of de-identified medical scans tobe received to train and/or retrain the model. For example, the imagetype parameters can correspond to one or more scan categories, canindicate scan classifier data 420, can indicate one or more scanmodalities, one or more anatomical regions, a date range, and/or otherparameters. The image type parameters can be determined by the centralserver system based on training parameters 620 determined for thecorresponding inference function to be trained, and/or based oncharacteristics of a new and/or existing scan category corresponding tothe inference function to be trained. The image type parameters can besent to the medical picture archive integration system 2600, and arequest such as a DICOM query can be sent to the medical picture archivesystem 2620, via transmitter 2604, that indicates the image typeparameters. For example, the processing system 2682 can be utilized togenerate the DICOM query based on the image type parameters receivedfrom the central server system 2640. The medical picture archive systemcan automatically transmit one or more DICOM images to the medicalpicture archive integration system in response to determining that theone or more DICOM images compares favorably to the image typeparameters. The DICOM images received in response can be de-identifiedby the de-identification system 2608. In some embodiments, thede-identified medical scans can be transmitted directly to the centralserver system 2640, for example, without generating annotation data.

The central server system can generate the new and/or updated modelparameters by training on the received set of de-identified medicalscans, and can transmit the new and/or updated model parameters to thede-identified storage system. If the model parameters correspond to anew inference function for a new scan category, the medical picturearchive integration system 2600 can generate a request, such as a DICOMquery, for transmission to the medical picture archive system indicatingthat incoming scans corresponding to image type parameters correspondingto the new scan category be sent to the medical picture archiveintegration system. The annotating system can update the set ofinference functions to include the new inference function, and theannotating system can select the new inference function from the set ofinference functions for subsequently generated de-identified medicalscans by the de-identification system by determining each of thesede-identified medical scans indicate the corresponding DICOM imagecorresponds to the new scan category. The new model parameters can beutilized to perform the new inference function on each of thesede-identified medical scans to generate corresponding annotation data,and an annotated DICOM file corresponding to each of these de-identifiedmedical scans can be generated for transmission to the medical picturearchive system via the transmitter.

In some embodiments, the central server system 2640 receives a pluralityof de-identified medical scans from a plurality of medical picturearchive integration system 2600, for example, each installed at aplurality of different hospitals or other medical entities, via thenetwork 2630. The central server system can generate training sets byintegrating de-identified medical scans from some or all of theplurality of medical picture archive integration systems 2600 to trainone or more inference functions and generate model parameters. Theplurality of medical picture archive integration systems 2600 canutilize the same set of inference functions or different sets ofinference functions. In some embodiments, the set of inference functionsutilized by the each of the plurality of medical picture archive systems2620 are trained on different sets of training data. For example, thedifferent sets of training data can correspond to the set ofde-identified medical scans received from the corresponding medicalpicture archive integration system 2600.

In some embodiments, the medical scan diagnosing system 108 can beutilized to implement the annotating system 2612, where thecorresponding subsystem processing device 235 and subsystem memorydevice 245 of the medical scan diagnosing system 108 are utilized toimplement the processing system 2682 and the memory 2684, respectively.Rather than receiving the medical scans via the network 150 as discussedin conjunction with FIG. 6A, the medical scan diagnosing system 108 canperform a selected medical scan inference function 1105 on an incomingde-identified medical scan generated by the de-identification system2608 and/or retrieved from the de-identified image storage system 2610.Memory 2684 can store the set of medical scan inference functions 1105,each corresponding to a scan category 1120, where the inference functionis selected from the set based on determining the scan category of thede-identified medical scan and selecting the corresponding inferencefunction. The processing system 2682 can perform the selected inferencefunction 1105 to generate the inference data 1110, which can be furtherutilized by the annotating system 2612 to generate the annotated DICOMfile for transmission back to the medical picture archive system 2620.New medical scan inference functions 1105 can be added to the set whencorresponding model parameters are received from the central serversystem. The remediation step 1140 can be performed locally by theannotating system 2612 and/or can be performed by the central serversystem 2640 by utilizing one or more de-identified medical scans andcorresponding annotation data sent to the central server system 2640.Updated model parameters can be generated by the central server system2640 and sent to the medical picture archive integration system 2600 asa result of performing the remediation step 1140.

The central server system 2640 can be implemented by utilizing one ormore of the medical scan subsystems 101, such as the medical scan imageanalysis system 112 and/or the medical scan diagnosing system 108, toproduce model parameters for one or more inference functions. Thecentral server system can store or otherwise communicate with a medicalscan database 342 that includes the de-identified medical scans and/orannotation data received from one or more medical picture archiveintegration systems 2600. Some or all entries of the medical scandatabase 342 can be utilized to as training data to produce modelparameters for one or more inference functions. These entries of themedical scan database 342 can be utilized by other subsystems 101 asdiscussed herein. For example, other subsystems 101 can utilize thecentral server system 2640 to fetch medical scans and/or correspondingannotation data that meet specified criteria. The central server system2640 can query the medical picture archive integration system 2600 basedon this criteria, and can receive de-identified medical scans and/orannotation data in response. This can be sent to the requestingsubsystem 101 directly and/or can be added to the medical scan database342 or another database of the database storage system 140 for access bythe requesting subsystem 101.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a user database 344 storing user profileentries corresponding to each of a plurality of medical entities thateach utilize a corresponding one of a plurality of medical picturearchive integration systems 2600. For example, basic user datacorresponding to the medical entity can be stored as basic user data, anumber of scans or other consumption information indicating usage of oneor more inference functions by corresponding medical picture archiveintegration system can be stored as consumption usage data, and/or anumber of scans or other contribution information indicatingde-identified scans sent to the central server system as training datacan be stored as contribution usage data. The user profile entry canalso include inference function data, for example, with a list of modelparameters or function identifiers, such as medical scan analysisfunction identifiers 357, of inference functions currently utilized bythe corresponding medical picture archive integration system 2600. Theseentries of the user database 344 can be utilized by other subsystems 101as discussed herein.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a medical scan analysis function database346 to store model parameters, training data, or other information forone or more inference functions as medical scan analysis functionentries 356. In some embodiments, model parameter data 623 can indicatethe model parameters and function classifier data 610 can indicate thescan category of inference function entries. In some embodiments, themedical scan analysis function entry 356 can further include usageidentifying information indicating a medical picture archive integrationsystem identifier, medical entity identifier, and/or otherwiseindicating which medical archive integration systems and/or medicalentities have received the corresponding model parameters to utilize theinference function corresponding to the medical scan analysis functionentry 356. These entries of the medical scan analysis function database346 can be utilized by other subsystems 101 as discussed herein.

In some embodiments, the de-identification function is a medical scananalysis function, for example, with a corresponding medical scananalysis function entry 356 in the medical scan analysis functiondatabase 346. In some embodiments, the de-identification function istrained by the central server system 2640. For example, the centralserver system 2640 can send de-identification function parameters to themedical picture archive integration system 2600 for use by thede-identification system 2608. In embodiments with a plurality ofmedical picture archive integration systems 2600, each of the pluralityof medical picture archive integration systems 2600 can utilize the sameor different de-identification functions. In some embodiments, thede-identification function utilized by the each of the plurality ofmedical picture archive integration systems 2600 are trained ondifferent sets of training data. For example, the different sets oftraining data can correspond to each different set of de-identifiedmedical scans received from each corresponding medical picture archiveintegration system 2600.

In some embodiments, as illustrated in FIGS. 8D-8F, the medical picturearchive integration system 2600 can further communicate with a reportdatabase 2625, such as a Radiology Information System (RIS), thatincludes a plurality of medical reports corresponding to the DICOMimages stored by the medical picture archive system 2620.

As shown in FIG. 8D, the medical picture archive integration system 2600can further include a receiver 2603 that receives report data,corresponding to the DICOM image, from report database 2625. The reportdatabase 2625 can be affiliated with the medical picture archive system2620 and can store report data corresponding to DICOM images stored inthe medical picture archive system. The report data of report database2625 can include PHI, and the report database 2625 can thus bedisconnected from network 2630.

The report data can include natural language text, for example,generated by a radiologist that reviewed the corresponding DICOM image.The report data can be used to generate the de-identified medical scan,for example, where the de-identification system 2608 performs a naturallanguage analysis function on the report data to identify patientidentifying text in the report data. The de-identification system 2608can utilize this patient identifying text to detect matching patientidentifiers in the DICOM image to identify the patient identifiers ofthe DICOM image and generate the de-identified medical scan. In someembodiments, the report data can be de-identified by obfuscating,hashing, removing, replacing with a fiducial, or otherwise anonymizingthe identified patient identifying text to generate de-identified reportdata.

The de-identified report data can be utilized by the annotating system2612, for example, in conjunction with the DICOM image, to generate theannotation data. For example, the annotating system 2612 can perform anatural language analysis function on the de-identified natural languagetext of the report data to generate some or all of the annotation data.In some embodiments, the de-identified report data is sent to thecentral server system, for example, to be used as training data forinference functions, for natural language analysis functions, for othermedical scan analysis functions, and/or for use by at least one othersubsystem 101. For example, other subsystems 101 can utilize the centralserver system 2640 to fetch medical reports that correspond toparticular medical scans or otherwise meet specified criteria. Thecentral server system 2640 can query the medical picture archiveintegration system 2600 based on this criteria, and can receivede-identified medical reports in response. This can be sent to therequesting subsystem 101 directly, can be added to the medical scandatabase 342, a de-identified report database, or another database ofthe database storage system 140 for access by the requesting subsystem101.

In some embodiments the medical picture archive integration system 2600can query the report database 2625 for the report data corresponding toa received DICOM image by utilizing a common identifier extracted fromthe DICOM image.

In some embodiments, the report data can correspond to a plurality ofDICOM images. For example, the report data can include natural languagetext describing a plurality of medical scans of a patient that caninclude multiple sequences, multiple modalities, and/or multiple medicalscans taken over time. In such embodiments, the patient identifying textand/or annotation data detected in the report data can also be appliedto de-identify and/or generate annotation data for the plurality ofDICOM images it describes. In such embodiments, the medical picturearchive integration system 2600 can query the medical picture archivesystem 2620 for one or more additional DICOM images corresponding to thereport data, and de-identified data and annotation data for theseadditional DICOM images can be generated accordingly by utilizing thereport data.

In some embodiments, as shown in FIG. 8E, the medical picture archivesystem 2620 communicates with the report database 2625. The medicalpicture archive system 2620 can request the report data corresponding tothe DICOM image from the report database 2625, and can transmit thereport data to the medical picture archive integration system 2600 via aDICOM communication protocol for receipt via receiver 2602. The medicalpicture archive system 2620 can query the report database 2625 for thereport data, utilizing a common identifier extracted from thecorresponding DICOM image, in response to determining to send thecorresponding DICOM image to the medical picture archive integrationsystem 2600.

FIG. 8F presents an embodiment where report data is generated by theannotating system 2612 and is transmitted, via a transmitter 2605, tothe report database 2625, for example via a DICOM communication protocolor other protocol recognized by the report database 2625. In otherembodiments, the report data is instead transmitted via transmitter 2604to the medical picture archive system 2620, and the medical picturearchive system 2620 transmits the report data to the report database2625.

The report data can be generated by the annotating system 2612 as outputof performing the inference function on the de-identified medical scan.The report data can include natural language text data 448 generatedautomatically based on other diagnosis data 440 such as abnormalityannotation data 442 determined by performing the inference function, forexample, by utilizing a medical scan natural language generatingfunction trained by the medical scan natural language analysis system114. The report data can be generated instead of, or in addition to, theannotated DICOM file.

FIG. 9 presents a flowchart illustrating a method for execution by amedical picture archive integration system 2600 that includes a firstmemory and a second memory that store executional instructions that,when executed by at least one first processor and at least one secondprocessor, respectfully, cause the medical picture archive integrationsystem to perform the steps below. In various embodiments, the firstmemory and at least one first processor are implemented by utilizing,respectfully, the memory 2654 and processing system 2652 of FIG. 8B. Invarious embodiments, the second memory is implemented by utilizing thememory 2674 and/or the memory 2684 of FIG. 8B. In various embodiments,the at least one second processor is implemented by utilizing theprocessing system 2682 of FIG. 8B.

Step 2702 includes receiving, from a medical picture archive system viaa receiver, a first DICOM image for storage in the first memory,designated for PHI, where the first DICOM image includes at least onepatient identifier. Step 2704 includes performing, via at least onefirst processor coupled to the first memory and designated for PHI, ade-identification function on the first DICOM image to identify the atleast one patient identifier and generate a first de-identified medicalscan that does not include the at least one patient identifier.

Step 2706 includes storing the first de-identified medical scan in asecond memory that is separate from the first memory. Step 2708 includesreceiving, via a network interface communicating with a network thatdoes not include the medical picture archive system, first modelparameters from a central server.

Step 2710 includes retrieving the first de-identified medical scan fromthe second memory. Step 2712 includes utilizing the first modelparameters to perform a first inference function on the firstde-identified medical scan to generate first annotation data via atleast one second processor that is different from the at least one firstprocessor. Step 2714 includes generating, via the at least one secondprocessor, a first annotated DICOM file for transmission to the medicalpicture archive system via a transmitter, where the first annotatedDICOM file includes the first annotation data and further includes anidentifier that indicates the first DICOM image. In various embodiments,the first annotated DICOM file is a DICOM presentation state file.

In various embodiments, the second memory further includes operationalinstructions that, when executed by the at least one second processor,further cause the medical picture archive integration system to retrievea second de-identified medical scan from the de-identified image storagesystem, where the second de-identified medical scan was generated by theat least one first processor by performing the de-identificationfunction on a second DICOM image received from the medical picturearchive system. The updated model parameters are utilized to perform thefirst inference function on the second de-identified medical scan togenerate second annotation data. A second annotated DICOM file isgenerated for transmission to the medical picture archive system via thetransmitter, where the second annotated DICOM file includes the secondannotation data and further includes an identifier that indicates thesecond DICOM image.

In various embodiments, the second memory stores a plurality ofde-identified medical scans generated by the at least one firstprocessor by performing the de-identification function on acorresponding plurality of DICOM images received from the medicalpicture archive system via the receiver. The plurality of de-identifiedmedical scans is transmitted to the central server via the networkinterface, and the central server generates the first model parametersby performing a training function on training data that includes theplurality of de-identified medical scans.

In various embodiments, the central server generates the first modelparameters by performing a training function on training data thatincludes a plurality of de-identified medical scans received from aplurality of medical picture archive integration systems via thenetwork. Each of the plurality of medical picture archive integrationsystems communicates bidirectionally with a corresponding one of aplurality of medical picture archive systems, and the plurality ofde-identified medical scans corresponds to a plurality of DICOM imagesstored by the plurality of medical picture archive integration systems.

In various embodiments, the first de-identified medical scan indicates ascan category of the first DICOM image. The second memory further storesoperational instructions that, when executed by the at least one secondprocessor, further cause the medical picture archive integration systemto select the first inference function from a set of inference functionsbased on the scan category. The set of inference functions correspondsto a set of unique scan categories that includes the scan category. Invarious embodiments, each unique scan category of the set of unique scancategories is characterized by one of a plurality of modalities and oneof a plurality of anatomical regions.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,further cause the medical picture archive integration system to receivea plurality of DICOM image data from the medical picture archive systemvia the receiver for storage in the first memory in response to a querytransmitted to the medical picture archive system via the transmitter.The query is generated by the medical picture archive integration systemin response to a request indicating a new scan category received fromthe central server via the network. The new scan category is notincluded in the set of unique scan categories, and the plurality ofDICOM image data corresponds to the new scan category. Thede-identification function is performed on the plurality of DICOM imagedata to generate a plurality of de-identified medical scans fortransmission to the central server via the network.

The second memory further stores operational instructions that, whenexecuted by the at least one second processor, further cause the medicalpicture archive integration system to receive second model parametersfrom the central server via the network for a new inference functioncorresponding to the new scan category. The set of inference functionsis updated to include the new inference function. The secondde-identified medical scan is retrieved from the first memory, where thesecond de-identified medical scan was generated by the at least onefirst processor by performing the de-identification function on a secondDICOM image received from the medical picture archive system. The newinference function is selected from the set of inference functions bydetermining the second de-identified medical scan indicates the secondDICOM image corresponds to the new scan category. The second modelparameters are utilized to perform the new inference function on thesecond de-identified medical scan to generate second annotation data. Asecond annotated DICOM file is generated for transmission to the medicalpicture archive system via the transmitter, where the second annotatedDICOM file includes the second annotation data and further includes anidentifier that indicates the second DICOM image.

In various embodiments, the medical picture archive integration systemgenerates parameter data for transmission to the medical picture archivesystem that indicates the set of unique scan categories. The medicalpicture archive system automatically transmits the first DICOM image tothe medical picture archive integration system in response todetermining that the first DICOM image compares favorably to one of theset of unique scan categories.

In various embodiments, the second memory further stores operationalinstructions that, when executed by the at least one second processor,cause the medical picture archive integration system to generate anatural language report data is based on the first annotation data andto transmit, via a second transmitter, the natural language report datato a report database associated with the medical picture archiveintegration system, where the natural language report data includes anidentifier corresponding to the first DICOM image.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,cause the medical picture archive integration system to receive, via asecond receiver, a natural language report corresponding to the firstDICOM image from the report database. A set of patient identifying textincluded in the natural language report are identified. Performing thede-identification function on the first DICOM image includes searchingthe first DICOM image for the set of patient identifying text toidentify the at least one patient identifier.

In various embodiments, the first memory is managed by a medical entityassociated with the medical picture archive system. The medical picturearchive integration system is located at a first geographic sitecorresponding to the medical entity, and the central server is locatedat a second geographic site. In various embodiments, the first memory isdecoupled from the network to prevent the first DICOM image thatincludes the at least one patient identifier from being communicated viathe network. In various embodiments, the medical picture archive systemis a Picture Archive and Communication System (PACS) server, and thefirst DICOM image is received in response to a query sent to the medicalpicture archive system by the transmitter in accordance with a DICOMcommunication protocol.

FIG. 10A presents an embodiment of a de-identification system 2800. Thede-identification system 2800 can be utilized to implement thede-identification system 2608 of FIGS. 8A-8F. In some embodiments, thede-identification system 2800 can be utilized by other subsystems tode-identify image data, medical report data, private fields of medicalscan entries 352 such as patient identifier data 431, and/or otherprivate fields stored in databases of the database memory device 340.

The de-identification system can be operable to receive, from at leastone first entity, a medical scan and a medical report corresponding tothe medical scan. A set of patient identifiers can be identified in asubset of fields of a header of the medical scan. A header anonymizationfunction can be performed on each of the set of patient identifiers togenerate a corresponding set of anonymized fields. A de-identifiedmedical scan can be generated by replacing the subset of fields of theheader of the medical scan with the corresponding set of anonymizedfields.

A subset of patient identifiers of the set of patient identifiers can beidentified in the medical report by searching text of the medical reportfor the set of patient identifiers. A text anonymization function can beperformed on the subset of patient identifiers to generate correspondinganonymized placeholder text for each of the subset of patientidentifiers. A de-identified medical report can be generated byreplacing each of the subset of patient identifiers with thecorresponding anonymized placeholder text. The de-identified medicalscan and the de-identified medical report can be transmitted to a secondentity via a network.

As shown in FIG. 10A, the de-identification system 2800 can include atleast one receiver 2802 operable to receive medical scans, such asmedical scans in a DICOM image format. The at least one receiver 2802 isfurther operable to receive medical reports, such as report data 449 orother reports containing natural language text diagnosing, describing,or otherwise associated the medical scans received by thede-identification system. The medical scans and report data can bereceived from the same or different entity, and can be received by thesame or different receiver 2802 in accordance with the same or differentcommunication protocol. For example, the medical scans can be receivedfrom the medical picture archive system 2620 of FIGS. 8A-8F and thereport data can be received from the report database 2625 of FIGS.8D-8F. In such embodiments, the receiver 2802 can be utilized toimplement the receiver 2602 of FIG. 8B.

The de-identification system 2800 can further include a processingsystem 2804 that includes at least one processor, and a memory 2806. Thememory 2806 can store operational instructions that, when executed bythe processing system, cause the de-identification system to perform atleast one patient identifier detection function on the received medicalscan and/or the medical report to identify a set of patient identifiersin the medical scan and/or the medical report. The operationalinstructions, when executed by the processing system, can further causethe de-identification system to perform an anonymization function on themedical scan and/or the medical report to generate a de-identifiedmedical scan and/or a de-identified medical report that do not includethe set of patient identifiers found in performing the at least onepatient identifier detection function. Generating the de-identifiedmedical scan can include generating a de-identified header andgenerating de-identified image data, where the de-identified medicalscan includes both the de-identified header and the de-identified imagedata. The memory 2806 can be isolated from Internet connectivity, andcan be designated for PHI.

The de-identification system 2800 can further include at least onetransmitter 2808, operable to transmit the de-identified medical scanand de-identified medical report. The de-identified medical scan andde-identified medical report can be transmitted back to the same entityfrom which they were received, respectively, and/or can be transmittedto a separate entity. For example, the at least one transmitter cantransmit the de-identified medical scan to the de-identified imagestorage system 2610 of FIGS. 8A-8F and/or can transmit the de-identifiedmedical scan to central server system 2640 via network 2630 of FIGS.8A-8F. In such embodiments, the transmitter 2808 can be utilized toimplement the interface 2655 of FIG. 8B. The receiver 2802, processingsystem 2804, memory 2806, and/or transmitter 2808 can be connected viabus 2810.

Some or all of the at least one patient identifier detection functionand/or at least one anonymization function as discussed herein can betrained and/or implemented by one or subsystems 101 in the same fashionas other medical scan analysis functions discussed herein, can be storedin medical scan analysis function database 346 of FIG. 3, and/or canotherwise be characterized by some or all fields of a medical scananalysis function entry 356 of FIG. 5.

The de-identification system 2800 can perform separate patientidentifier detection functions on the header of a medical report and/ormedical scan, on the text data of the medical report, and/or on theimage data of the medical scan, such as text extracted from the imagedata of the medical scan. Performance of each of these functionsgenerates an output of its own set of identified patient identifiers.Combining these sets of patient identifiers yields a blacklist term set.A second pass of the header of a medical report and/or medical scan, onthe text data of the medical report, and/or on the image data of themedical scan that utilizes this blacklist term set can catch any termsthat were missed by the respective patient identifier detectionfunction, and thus, the outputs of these multiple identificationprocesses can support each other. For example, some of the data in theheaders will be in a structured form and can thus be easier to reliablyidentify. This can be exploited and used to further anonymize theseidentifiers when they appear in free text header fields, report data,and/or in the image data of the medical scan. Meanwhile, unstructuredtext in free text header fields, report data, and/or image data of themedical scan likely includes pertinent clinical information to bepreserved in the anonymization process, for example, so it can beleveraged by at least one subsystems 101 and/or so it can be leveragedin training at least one medical scan analysis function.

At least one first patient identifier detection function can includeextracting the data in a subset of fields of a DICOM header, or anotherheader or other metadata of the medical scan and/or medical report witha known type that corresponds to patient identifying data. For example,this patient identifying subset of fields can include a name field, apatient ID number field or other unique patient identifier field, a datefield, a time field, an age field, an accession number field, SOPinstance UID, and/or other fields that could be utilized to identify thepatient and/or contain private information. A non-identifying subset offields of the header can include hospital identifiers, machine modelidentifiers, and/or some or all fields of medical scan entry 352 that donot correspond to patient identifying data. The patient identifyingsubset of fields and the non-identifying subset of fields can bemutually exclusive and collectively exhaustive with respect to theheader. The at least one patient identifier function can includegenerating a first set of patient identifiers by ignoring thenon-identifying subset of fields and extracting the entries of thepatient identifying subset of fields only. This first set of patientidentifiers can be anonymized to generate a de-identified header asdiscussed herein.

In some embodiments, at least one second patient identifier detectionfunction can be performed on the report data of the medical report. Theat least one second patient identifier detection function can includeidentifying patient identifying text in the report data by performing anatural language analysis function, for example, trained by the medicalscan natural language analysis system 114. For example, the at least onesecond patient identifier detection function can leverage the knownstructure of the medical report and/or context of the medical report. Asecond set of patient identifiers corresponding to the patientidentifying text can be determined, and the second set of patientidentifiers can be anonymized to generate a de-identified medicalreport. In some embodiments, a de-identified medical report includesclinical information, for example, because the portion of the originalmedical report that includes the clinical information was deemed to befree of patient identifying text and/or because the portion of theoriginal medical report that includes the clinical information wasdetermined to include pertinent information to be preserved.

In some embodiments, the medical report includes image datacorresponding to freehand or typed text. For example the medical reportcan correspond to a digitized scan of original freehand text written bya radiologist or other medical professional. In such embodiments, thepatient identifier detection function can first extract the text fromthe freehand text in the image data to generate text data before the atleast one second patient identifier detection function is performed onthe text of the medical report to generate the second set of patientidentifiers.

In some embodiments, the at least one second patient identifierdetection function can similarly be utilized to identify patientidentifying text in free text fields and/or unstructured text fields ofa DICOM header and/or other metadata of the medical scan and/or medicalreport data by performing a natural language analysis function, forexample, trained by the medical scan natural language analysis system114. A third set of patient identifiers corresponding to this patientidentifying text of the free text and/or unstructured header fields canbe determined, and the third set of patient identifiers can beanonymized to generate de-identified free text header field and/orunstructured header fields. In some embodiments, a de-identified freetext header field and/or unstructured header field includes clinicalinformation, for example, because the portion of the originalcorresponding header field that includes the clinical information wasdeemed to be free of patient identifying text and/or because the portionof the original corresponding header field that includes the clinicalinformation was determined to include pertinent information to bepreserved.

Patient identifiers can also be included in the image data of themedical scan itself. For example, freehand text corresponding to apatient name written on a hard copy of the medical scan beforedigitizing can be included in the image data, as discussed inconjunction with FIG. 10B. Other patient identifiers, such asinformation included on a patient wristband or other identifyinginformation located on or within the vicinity of the patient may havebeen captured when the medical scan was taken, and can thus be includedin the image. At least one third patient identifier detection functioncan include extracting text from the image data and/or detectingnon-text identifiers in the image data by performing a medical scanimage analysis function, for example, trained by the medical scan imageanalysis system 112. For example, detected text that corresponds to animage location known to include patient identifiers, detected text thatcorresponds to a format of a patient identifier, and/or or detected textor other image data determined to correspond to a patient identifier canbe identified. The at least one third patient identifier detectionfunction can further include identifying patient identifying text in thetext extracted from the image data by performing the at least one secondpatient identifier detection function and/or by performing a naturallanguage analysis function. A fourth set of patient identifierscorresponding to patient identifying text or other patient identifiersdetected in the image data of the medical scan can be determined, andthe fourth set of patient identifiers can be anonymized in the imagedata to generate de-identified image data of the medical scan asdescribed herein. In particular, the fourth set of patient identifierscan be detected in a set of regions of image data of the medical scan,and the set of regions of the image data can be anonymized.

In some embodiments, only a subset of the patient identifier detectionfunctions described herein are performed to generate respective sets ofpatient identifiers for anonymization. In some embodiments, additionalpatient identifier detection functions can be performed on the medicalscan and/or medical report to determine additional respective sets ofpatient identifiers for anonymization. The sets of patient identifiersoutputted by performing each patient identifier detection function canhave a null or non-null intersection. The sets of patient identifiersoutputted by performing each patient identifier function can have nullor non-null set differences.

Cases where the sets of patient identifiers have non-null setdifferences can indicate that a patient identifier detected by onefunction may have been missed by another function. The combined set ofpatient identifiers, for example, generated as the union of the sets ofsets of patient identifiers outputted by performing each patientidentifier function, can be used to build a blacklist term set, forexample, stored in memory 2806. The blacklist term set can designate thefinal set of terms to be anonymized. A second pass of header data,medical scans, medical reports, and/or any free text extracted from theheader data, the medical scan, and/or the medical report can beperformed by utilizing the blacklist term set to flag terms foranonymization that were not caught in performing the respective at leastone patient identifier detection function. For example, performing thesecond pass can include identifying at least one patient identifier ofthe blacklist term set in the header, medical report, and/or image dataof the medical scan. This can include by searching correspondingextracted text of the header, medical report, and/or image data forterms included in blacklist term set and/or by determining if each termin the extracted text is included in the blacklist term set.

In some embodiments, at least one patient identifier is not detecteduntil the second pass is performed. Consider an example where a freetext field of a DICOM header included a patient name that was notdetected in performing a respective patient identifier detectionfunction on the free text field of the DICOM header. However, thepatient name was successfully identified in the text of the medicalreport in performing a patient identifier detection function on themedical report. This patient name is added to the blacklist term list,and is detected in a second pass of the free text field of the DICOMheader. In response to detection in the second pass, the patient name ofthe free text field of the DICOM header can be anonymized accordingly togenerate a de-identified free text field. Consider a further examplewhere the patient name is included in the image data of the medicalscan, but was not detected in performing a respective patient identifierdetection function on the free text field of the DICOM header. In thesecond pass, this patient name can be detected in at least one region ofimage data of the medical scan by searching the image data for theblacklist term set.

In some embodiments, performing some or all of the patient identifierdetection functions includes identifying a set of non-identifying terms,such as the non-identifying subset of fields of the header. Inparticular, the non-identifying terms can include terms identified asclinical information and/or other terms determined to be preserved. Thecombined set of non-identifying terms, for example, generated as theunion of the sets of sets of non-identifying outputted by performingeach patient identifier function, can be used to build a whitelist termset, for example, stored in memory 2806. Performing the second pass canfurther include identifying at least one non-identifying term of thewhitelist term set in the header, medical report, and/or image data ofthe medical scan, and determining not to anonymize, or to otherwiseignore, the non-identifying term.

In various embodiments, some or all terms of the whitelist term set canbe removed from the blacklist term set. In particular, at least one termpreviously identified as a patient identifier in performing one or morepatient identifier detection functions is determined to be ignored andnot anonymized in response to determining the term is included in thewhitelist term set. This can help ensure that clinically importantinformation is not anonymized, and is thus preserved in thede-identified medical scan and de-identified medical report.

In some embodiments, the second pass can be performed after each of thepatient identifier detection functions are performed. For example,performing the anonymization function can include performing this secondpass by utilizing the blacklist term set to determine the final set ofterms to be anonymized. New portions of text in header fields, notpreviously detected in generating the first set of patient identifiersor the third set of patient identifiers, can be flagged foranonymization by determining these new portions of text correspond toterms of the blacklist term set. New portions of text the medicalreport, not previously detected in generating in the second set ofpatient identifiers, can be flagged for anonymization by determiningthese new portions of text correspond to terms of the blacklist termset. New regions of the image data of the medical scan, not previouslydetected in generating the fourth set of patient identifiers, can beflagged for anonymization by determining these new portions of textcorrespond to terms of the blacklist term set.

In some embodiments, the blacklist term set is built as each patientidentifier detection function is performed, and performance ofsubsequent patient identifier detection functions includes utilizing thecurrent blacklist term set. For example, performing the second patientidentifier detection function can include identifying a first subset ofthe blacklist term set in the medical report by searching the text ofthe medical report for the blacklist term set and/or by determining ifeach term in the text of the medical report is included in the blacklistterm set. Performing the second patient identifier detection functioncan further include identifying at least one term in the medical reportthat is included in the whitelist term set, and determining to ignorethe term in response. The first subset can be anonymized to generate thede-identified medical report as discussed herein. New patientidentifiers not already found can be appended to the blacklist term set,and the updated blacklist term set can be applied to perform a secondsearch of the header and/or image data of the medical scan, and at leastone of the new patient identifiers can be identified in the header inthe second search of the header and/or in the image data in a secondsearch of the image data. These newly identified patient identifiers inthe header and/or image data are anonymized in generating thede-identified medical scan.

As another example, a second subset of the blacklist term set can bedetected in a set of regions of image data of the medical scan byperforming the medical scan image analysis function on image data of themedical scan, where the image analysis function includes searching theimage data for the set of patient identifiers. For example, the medicalscan image analysis function can include searching the image data fortext, and the second subset can include detected text that matches oneor more terms of the blacklist term set. In some embodiments, detectedtext that matches one or more terms of the whitelist term set can beignored. The second subset can be anonymized to generate de-identifiedimage data as discussed herein. New patient identifiers that aredetected can be appended to the blacklist term set, and the updatedblacklist term set can be applied to perform a second search of theheader and/or metadata of the medical scan, and/or can be applied toperform a second search of the medical report. At least one of the newpatient identifiers can be identified in the header as a result ofperforming the second search of the header and/or at least one of thenew patient identifiers can be identified medical report as a result ofperforming the second search of the medical report. These newlyidentified patient identifiers can be anonymized in the header alongwith the originally identified blacklist term set in generating thede-identified header, and/or can be anonymized in the medical reportalong with the originally identified first subset in generating thede-identified medical report.

In some embodiments, the memory 2806 further stores a global blacklist,for example, that includes a vast set of known patient identifyingterms. In some embodiments, the global blacklist is also utilized by atleast one patient identifier detection function and/or in performing thesecond pass to determine patient identifying terms for anonymization. Insome embodiments, the blacklist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal blacklist for use in performing the second pass and/or indetecting patient identifiers in subsequently received medical scansand/or medical reports.

Alternatively or in addition, the memory 2806 can further store a globalwhitelist, for example, that includes a vast set of terms that can beignored. In particular, the global whitelist can include clinical termsand/or other terms that are deemed beneficial to preserve that do notcorrespond to patient identifying information. In some embodiments, theglobal whitelist is utilized by at least one patient identifierdetection function and/or in performing the second pass to determineterms to ignore in the header, image data, and/or medical report. Insome embodiments, the whitelist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal whitelist for use in performing the second pass and/or inignoring terms in subsequently received medical scans and/or medicalreports.

Alternatively or in addition, the memory 2806 can further store a globalgraylist, for example, that includes ambiguous terms that could bepatient identifying terms in some contexts, but non-identifying terms inother contexts. For example, “Parkinson” could correspond to patientidentifying data if part of a patient name such as “John Parkinson”, butcould correspond to non-patient identifying data meant to be ignored andpreserved in the de-identified medical report and/or de-identifiedmedical scan if part of a diagnosis term such as “Parkinson's disease.”In some embodiments, the global graylist is also utilized in performingthe second pass and/or in performing at least one patient identifierdetection function to determine that a term is included in the graylist,and to further determine whether the term should be added to theblacklist term set for anonymization or whitelist term set to be ignoredby leveraging context of accompanying text, by leveraging known datatypes of a header field from which the term was extracted, by leveragingknown structure of the term, by leveraging known data types of alocation of the image data from which the term was extracted, and/or byleveraging other contextual information. In some embodiments, thegraylist term set can be updated based on blacklist and/or whitelistterm sets for a particular medical scan and corresponding medicalreport.

In some embodiments, the at least one anonymization function includes afiducial replacement function. For example, some or all of the blacklistterm set can be replaced with a corresponding, global fiducial in theheader, report data, and/or image data. In some embodiments, the globalfiducial can be selected from a set of global fiducials based on a typeof the corresponding patient identifier. Each patient identifierdetected in the header and/or medical report can be replaced with acorresponding one of the set of global text fiducials. Each patientidentifiers detected in the image data can be replaced with acorresponding one of the set of global image fiducials. For example, oneor more global image fiducials can overlay pixels of regions of theimage data that include the identifying patient data, to obfuscate theidentifying patient data in the de-identified image data.

The global text fiducials and/or global image fiducials can berecognizable by inference functions and/or training functions, forexample, where the global text fiducials and global image fiducials areignored when processed in a training step to train an inference functionand/or are ignored in an inference step when processed by an inferencefunction. Furthermore, the global text fiducials and/or global imagefiducials can be recognizable by a human viewing the header, medicalreport, and/or image data. For example, a radiologist or other medicalprofessional, upon viewing a header, medical report, and/or image data,can clearly identify the location of a patient identifier that wasreplaced by the fiducial and/or can identify the type of patientidentifier that was replaced by the fiducial.

As an example, the name “John Smith” can be replaced in a header and/ormedical report with the text “% PATIENT NAME %”, where the text “%PATIENT NAME %” is a global fiducial for name types of the header and/orthe text of medical reports. The training step and/or inference step ofmedical scan natural language analysis functions can recognize andignore text that matches “% PATIENT NAME %” automatically.

FIG. 10B illustrates an example of anonymizing patient identifiers inimage data of a medical scan. In this example, the name “John Smith” andthe date “May 4, 2010” is detected as freehand text in the originalimage data of a medical scan. The regions of the image data that includethe patient identifiers can each be replaced by global fiducial in theshape of a rectangular bar, or any other shape. As shown in FIG. 10B, afirst region corresponding to the location of “John Smith” in theoriginal image data is replaced by fiducial 2820 in the de-identifiedimage data, and a second region corresponding to the location of “May 4,2010” in the original image data is replaced by fiducial 2822 in thede-identified image data. The size, shape, and/or location of eachglobal visual fiducial can be automatically determined based on thesize, shape, and/or location of the region that includes the patientidentifier to minimize the amount of the image data that is obfuscated,while still ensuring the entirety of the text is covered. While notdepicted in FIG. 10B, the fiducial can be of a particular color, forexample, where pixels of the particular color are automaticallyrecognized by the training step and/or inference step of medical scanimage analysis functions to indicate that the corresponding region beignored, and/or where the particular color is not included in theoriginal medical scan and/or is known to not be included in any medicalscans. The fiducial can include text recognizable to human inspectionsuch as “% PATIENT NAME” and “% DATE” as depicted in FIG. 10B, and/orcan include a QR code, logo, or other unique symbol recognizable tohuman inspection and/or automatically recognizable by the training stepand/or inference step of medical scan image analysis functions toindicate that the corresponding region be ignored.

In some embodiments, other anonymization functions can be performed ondifferent ones of the patient identifying subset of fields to generatethe de-identified header, de-identified report data, and/orde-identified image data. For example, based on the type of identifyingdata of each field of the header, different types of headeranonymization functions and/or text anonymization functions can beselected and utilized on the header fields, text of the report, and/ortext extracted from the image data. A set of anonymization functions caninclude a shift function, for example, utilized to offset a date, timeor other temporal data by a determined amount to preserve absolute timedifference and/or to preserve relative order over multiple medical scansand/or medical reports of a single patient. FIG. 10B depicts an examplewhere the shift function is performed on the date detected in the imagedata to generate fiducial 2822, where the determined amount is 10 yearsand 1 month. The determined amount can be determined by thede-identification system randomly and/or pseudo-randomly for eachpatient and/or for each medical scan and corresponding medical report,ensuring the original date cannot be recovered by utilizing a knownoffset. In various embodiments, other medical scans and/or medicalreports are fetched for the same patient by utilizing a patient IDnumber or other unique patient identifier of the header. These medialscans and reports can be anonymized as well, where the dates and/ortimes detected in these medical scans and/or medical reports offset bythe same determined amount, randomized or pseudo-randomized forparticular patient ID number, for example, based on performing a hashfunction on the patient ID number.

The set of anonymization functions can include at least one hashfunction, for example utilized to hash a unique patient ID such as apatient ID number, accession number, and/or SOP instance UID of theheader and/or text. In some embodiments, the hashed SOP instance UID,accession number, and/or patient ID number are prepended with a uniqueidentifier, stored in a database of the memory 2806 and/or shared withthe entities to which the de-identified medical scans and/or medicalreports are transmitted, so that de-identified medical scans and theircorresponding de-identified medical reports can be linked and retrievedretroactively. Similarly, longitudinal data can be preserved as multiplemedical scans and/or medical reports of the same patient will beassigned the same hashed patient ID.

The set of anonymization functions can further include at least onemanipulator function for some types of patient identifiers. Some valuesof header fields and/or report text that would normally not beconsidered private information can be considered identifying patientdata if they correspond to an outlier value or other rare value thatcould then be utilized to identify the corresponding patient from a verysmall subset of possible options. For example, a patient age over 89could be utilized to determine the identity of the patient, for example,if there are very few patients over the age of 89. To prevent suchcases, in response to determining that a patient identifier correspondsto an outlier value and/or in response to determining that a patientidentifier compares unfavorably to a normal-range threshold value, thepatient identifier can be capped at the normal-range threshold value orcan otherwise be manipulated. For example, a normal-range thresholdvalue corresponding to age can be set at 89, and generating ade-identified patient age can include capping patient ages that arehigher than 89 at 89 and/or can include keeping the same value forpatient ages that are less than or equal to 89.

In some embodiments, the de-identified header data is utilized toreplace the corresponding first subset of patient identifiers detectedin the medical report with text of the de-identified header fields. Inother embodiments, a set of text anonymization functions includes aglobal text fiducial replacement function, shift function, a hashfunction, and/or manipulator functions that anonymize the correspondingtypes of patient identifiers in the medical report separately.

In some embodiments where the image data of a medical scan includes ananatomical region corresponding to a patient's head, the image data mayinclude an identifying facial structure and/or facial features thatcould be utilized to determine the patient's identity. For example, adatabase of facial images, mapped to a corresponding plurality of peopleincluding the patient, could be searched and a facial recognitionfunction could be utilized to identify the patient in the database.Thus, facial structure included in the image data can be consideredpatient identifying data.

To prevent this problem and maintain patient privacy, thede-identification system can further be implemented to perform facialobfuscation for facial structure detected in medical scans. At least oneregion of the image data that includes identifying facial structure canbe determined by utilizing a medical image analysis function. Forexample, the medical image analysis function can include a facialdetection function that determines the regions of the image data thatinclude identifying facial structure based on searching the image datafor pixels with a density value that corresponds to facial skin, facialbone structure, or other density of an anatomical mass type thatcorresponds to identifying facial structure, and the facial obfuscationfunction can be performed on the identified pixels. Alternatively or inaddition, the facial detection function can determine the region basedon identifying at least one shape in the image data that corresponds toa facial structure.

The image obfuscation function can include a facial structureobfuscation function performed on the medical scan to generatede-identified image data that does not include identifying facialstructure. For example, the facial structure obfuscation function canmask, scramble, replace with a fiducial, or otherwise obfuscate thepixels of the region identified by the facial detection function. Insome embodiments, the facial structure obfuscation function can performa one-way function on the region that preserves abnormalities of thecorresponding portions of the image, such as nose fractures or facialskin legions, while still obfuscating the identifying facial structuresuch that the patient is not identifiable. For example, the pixels ofthe identifying facial structure can be altered such that they convergetowards a fixed, generic facial structure. In some embodiments, aplurality of facial structure image data of a plurality of patients canbe utilized to generate the generic facial structure, for example,corresponding to an average or other combination of the plurality offaces. For example, the pixels of the generic facial structure can beaveraged with, superimposed upon, or otherwise combined with the pixelsof the region of the image data identified by the facial detectionfunction in generating the de-identified image data.

In some embodiments, a hash function can be performed on an average ofthe generic facial structure and the identified facial structure of theimage data so that the generic facial structure cannot be utilized inconjunction with the resulting data of the de-identified image data toreproduce the original, identifying facial structure. In suchembodiments, the hash function can alter the pixel values while stillpreserving abnormalities. In some embodiments, a plurality of random,generic facial structures can be generated by utilizing the plurality offacial structure image data, for example, where each if the plurality offacial structure image data are assigned a random or pseudo-randomweight in an averaging function utilized to create the generic facialstructure, where a new, random or pseudo-random set of weights aregenerated each time the facial structure obfuscation function isutilized to create a new, generic facial structure to be averaged withthe identified facial structure in creating the de-identified image datato ensure the original identifying facial structure cannot be extractedfrom the resulting de-identified image data.

While facial obfuscation is described herein, similar techniques can beapplied in a similar fashion to other anatomical regions that aredetermined to include patient identifiers and/or to other anatomicalregions that can be utilized to extract patient identifying informationif not anonymized.

In some embodiments, the at least one receiver 2802 is included in atleast one transceiver, for example, enabling bidirectional communicationbetween the medical picture archive system 2620 and/or the reportdatabase 2625. In such embodiments, the de-identification system 2800can generate queries to the medical picture archive system 2620 and/orthe report database 2625 for particular medical scans and/or medicalreports, respectively. In particular, if the medical scan and medicalreport are stored and/or managed by separate memories and/or separateentities, they may not be received at the same time. However, a linkingidentifier, such as DICOM identifiers in headers or metadata of themedical scan and/or medical report, such accession number, patient IDnumber, SOP instance UID, or other linking identifier that maps themedical scan to the medical report can be utilized to fetch a medicalreport corresponding to a received medical scan and/or to fetch amedical scan corresponding to a received medical report via a query sentutilizing the at least one transceiver. For example, in response toreceiving the medical scan from the medical picture archive system 2620,the de-identification system can extract a linking identifier from aDICOM header of the medical scan, and can query the report database 2625for the corresponding medical report by indicating the linkingidentifier in the query. Conversely, in response to receiving themedical report from the report database 2625, the de-identificationsystem can extract the linking identifier from a header, metadata,and/or text body of the medical report, and can query the medicalpicture archive system 2620 for the corresponding medical scan byindicating the linking identifier in the query. In some embodiments, amapping of de-identified medical scans to original medical scans, and/ora mapping of de-identified medical reports to original medical reportscan be stored in memory 2806. In some embodiments, linking identifierssuch as patient ID numbers can be utilized to fetch additional medicalscans, additional medical reports, or other longitudinal datacorresponding to the same patient.

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system 2800 that stores executional instructions that,when executed by at least one processor, cause the de-identification toperform the steps below.

Step 2902 includes receiving from a first entity, via a receiver, afirst medical scan and a medical report corresponding to the medicalscan. Step 2904 includes identifying a set of patient identifiers in asubset of fields of a first header of the first medical scan. Step 2906includes performing a header anonymization function on each of the setof patient identifiers to generate a corresponding set of anonymizedfields. Step 2908 includes generating a first de-identified medical scanby replacing the subset of fields of the first header of the firstmedical scan with the corresponding set of anonymized fields. Step 2910includes identifying a first subset of patient identifiers of the set ofpatient identifiers in the medical report by searching text of themedical report for the set of patient identifiers. Step 2912 includesperforming a text anonymization function on the first subset of patientidentifiers to generate corresponding anonymized placeholder text foreach of the first subset of patient identifiers. Step 2914 includesgenerating a de-identified medical report by replacing each of the firstsubset of patient identifiers with the corresponding anonymizedplaceholder text. Step 2916 includes transmitting, via a transmitter,the de-identified first medical scan and the de-identified medicalreport to a second entity via a network.

In various embodiments, the medical scan is received from a PictureArchive and Communication System (PACS), where the medical report isreceived from a Radiology Information System (RIS), and where the firstde-identified medical scan and the de-identified medical report aretransmitted to a central server that is not affiliated with the PACS orthe RIS. In various embodiments, first medical scan and the medicalreport are stored in a first memory for processing. The first memory isdecoupled from the network to prevent the set of patient identifiersfrom being communicated via the network. The first de-identified medicalscan and the de-identified medical report are stored in a second memorythat is separate from the first memory. The first de-identified medicalscan and the de-identified medical report are fetched from the secondmemory for transmission to the second entity.

In various embodiments, the header anonymization function performed oneach of the set of patient identifiers is selected from a plurality ofheader anonymization functions based on one of a plurality of identifiertypes of the corresponding one of the subset of fields. In variousembodiments, the plurality of identifier types includes a date type. Ashift function corresponding to the date type is performed on a firstdate of the first header to generate the first de-identified medicalscan, where the shift function includes offsetting the first date by adetermined amount. A second medical scan is received, via the receiver,that includes a second header. A unique patient ID of the first headermatches a unique patient ID of the second header. The shift function isperformed on a second date of the second header by offsetting the seconddate by the determined amount to generate a second de-identified medicalscan. The second de-identified medical scan is transmitted to the secondentity via the network.

In various embodiments, the plurality of identifier types includes aunique patient ID type. A hash function corresponding the unique patientID type is performed on the unique patient ID of the first header togenerate the first de-identified medical scan. The hash function isperformed on the unique patient ID of the second header to generate thesecond de-identified medical scan. An anonymized unique patient ID fieldof the first de-identified medical scan matches an anonymized uniquepatient ID field of the second de-identified medical scan as a result ofthe unique patient ID of the first header matching the unique patient IDof the second header.

In various embodiments, the plurality of identifier types includes alinking identifier type that maps the medical scan to the medicalreport. A hash function corresponding to the linking identifier type isperformed on a linking identifier of the first header to generate ahashed linking identifier. A linking identifier field of the firstde-identified medical scan includes the hashed linking identifier.Performing the text anonymization function on the first subset ofpatient identifiers includes determining one of the first subset ofpatient identifiers corresponds to linking identifier text andperforming the hash function on the one of the first subset of patientidentifiers to generate the hashed linking identifier, where thede-identified medical report includes the hashed linking identifier.

In various embodiments, a second subset of patient identifiers of theset of patient identifiers is identified in a set of regions of imagedata of the medical scan by performing an image analysis function onimage data of the medical scan. The image analysis function includessearching the image data for the set of patient identifiers. Anidentifier type is determined for each of the second subset of patientidentifiers. One of a plurality of image fiducials is selected for eachof the second subset of patient identifiers based on the identifiertype. De-identified image data is generated, where a set of regions ofthe de-identified image data, corresponding to the set of regions of theimage data, includes the one of the plurality of image fiducials toobfuscate each of the second subset of patient identifiers. Generatingthe first de-identified medical scan further includes replacing theimage data of the medical scan with the de-identified image data.

In various embodiments, a new patient identifier is identified in themedical report by performing a natural language analysis function on themedical report, where new patient identifier is not included in the setof patient identifiers. The set of patient identifiers is updated toinclude the new patient identifier prior to searching the image data ofthe medical scan for the set of patient identifiers, and the secondsubset of patient identifiers includes the new patient identifier.

In various embodiments, the memory further stores a global identifierblacklist. The natural language analysis function includes searching themedical report for a plurality of terms included in the globalidentifier blacklist to identify the new patient identifier. In variousembodiments, the de-identification system determines that the globalidentifier blacklist does not include one of the set of patientidentifiers, and the global identifier blacklist is updated to includethe one of the set of patient identifiers.

In various embodiments, performing the image analysis function furtherincludes identifying a new patient identifier in the image data, wherenew patient identifier is not included in the set of patientidentifiers. Identifying text is extracted from a region of the imagedata corresponding to the new patient identifier. The new patientidentifier is identified in the medical report by searching text of themedical report for the identifying text. The text anonymization functionis performed on new patient identifier to generate anonymizedplaceholder text for the new patient identifier. Generating thede-identified medical report further includes replacing the identifyingtext with the anonymized placeholder text for the new patientidentifier.

In various embodiments, generating the de-identified image data furtherincludes detecting an identifying facial structure in the image data ofthe medical scan. Generating the de-identified image data includesperforming a facial structure obfuscation function on the image data,and where the de-identified image data does not include the identifyingfacial structure.

FIGS. 12A-12G illustrate an embodiments of a medical scan hierarchicallabeling system 3002. The medical scan hierarchical labeling system 3002can be utilized to generate structured labeling data for medical scansvia one or more client devices 120, based on user input to aninteractive interface displayed on a display device corresponding to theone or more client devices.

As shown in FIGS. 12A-12G, the medical scan hierarchical labeling system3002 can communicate bi-directionally, via network 150, with the medicalscan database 342, medical scan analysis function database 346, and/orother databases of the database storage system 140, with one or moreclient devices 120, and/or, while not shown in FIG. 12A, one or moresubsystems 101 of FIG. 1. In some embodiments, the medical scanhierarchical labeling system 3002 is an additional subsystem 101 of themedical scan processing system 100, implemented by utilizing thesubsystem memory device 245, subsystem processing device 235, and/orsubsystem network interface 265 of FIG. 2A. In some embodiments, themedical scan hierarchical labeling system 3002 is implemented byutilizing, or otherwise communicates with, the central server 2640. Forexample, some or all of the databases of the database storage system 140are populated with de-identified data generated by the medical picturearchive integration system 2600. In some embodiments, the medical scanhierarchical labeling system 3002 can receive de-identified medicalscans, annotation data, and/or reports directly from the medical picturearchive integration system 2600. For example, the medical scanhierarchical labeling system 3002 can request de-identified medicalscans, annotation data, and/or reports that match requested criteria,for example, corresponding to training set criteria. In someembodiments, some or all of the medical scan hierarchical labelingsystem 3002 is implemented by utilizing other subsystems 101 and/or isoperable to perform functions or other operations described inconjunction with one or more other subsystems 101. In some embodiments,the medical scan hierarchical labeling system 3002 is integrated withinand/or utilizes the medical scan assisted review system 102 and/or themedical scan annotator system 106.

As shown in FIG. 12A, the medical scan hierarchical labeling system 3002can store labeling application data 3020 of a labeling applicationassociated with the medical scan hierarchical labeling system 3002. Thelabeling application data 3020 can include a plurality of promptdecision trees. The plurality of prompt decision trees can include adiagnosis prompt decision tree 3022, a characterization prompt decisiontree 3024, and/or a localization prompt decision tree 3026. The labelingapplication data 3020 can be sent to the one or more client devices. Thelabeling application data can be stored as a client application asillustrated in FIG. 2A, stored in client memory device 240. The clientprocessing device 230 can execute operational instructions of thelabeling application data to enable the corresponding client device 120to run the labeling application. The labeling application can include aninteractive interface 3075 displayed on display device 270.

The client device can further receive a medical scan from the medicalscan for labeling, for example, as a transmission from the medical scanhierarchical labeling system 3002, fetched directly from the medicalscan database 342, and/or uploaded to the client device directly. Asshown in FIG. 12B, the client device can utilize the labelingapplication to generate labeling data for the medical scan fortransmission back to the medical scan hierarchical labeling system 3002.The labeling data can be mapped to the medical scan in the medical scandatabase, and can correspond to some or all fields of a medical scanentry 352, such as diagnosis data 440.

The medical scan database can correspond to a relational database and/ora database with a structured set of fields for the plurality of medicalscan entries. The labeling data generated via the labeling applicationcan correspond to the structured set of fields. In particular, each ofthe plurality of prompt decision trees can include leaf nodes thatcorrespond to the structured set of fields, and the labeling applicationcan display prompts to a user of the client device in accordance withthe plurality of prompt decision trees. The labeling application cangenerate labeling data that corresponds to leaf nodes of the pluralityof prompt decision tree, based on user input to the prompts indicatingselections from sets of selection options that correspond to subsets ofthe structured set of fields. Requiring that all labeling data adheresto a uniform structure with a discrete set of possibilities in thisfashion allows the labeling data to be consumable and easily utilized byother systems. For example, the labeling data can be utilized astraining data for one or more other subsystems 101 train a medical scananalysis function.

The labeling application can be utilized by users such as radiologistsand/or other labelers responsible for labeling and/or annotating medicalscans of the medical scan database with diagnosis data. The interactiveinterface 3075 can display image data of the medical scan to such a userin conjunction with a plurality of prompts to provide diagnosis,characterization, and/or localization labeling of the medical scan forat least one abnormality identified by the user. The plurality ofprompts can present a fixed set of differential diagnosis optionsallowing the user to select one or more of the differential diagnosisoptions corresponding to one or more abnormalities detected by the userin the medical scan. The plurality of prompts can present a fixed set ofcharacterization options to classify, describe, or otherwisecharacterize the one or more differential diagnoses identified in thefixed set of diagnosis data options. The plurality of prompts canpresent a fixed set of localization options to indicate a region ofinterest and/or specific anatomical location associated with the one ormore differential diagnoses identified in the fixed set of diagnosisdata options.

The fixed sets of diagnosis options, characterization option, and/orlocalization options can correspond to a fixed set of hierarchicaloptions, and can be characterized by a diagnosis prompt decision tree, acharacterization prompt decision tree, and/or a localization promptdecision tree, respectively. FIGS. 12C, 12D, and 12E illustrate examplesof a diagnosis prompt decision tree 3022, a characterization promptdecision tree 3024, and/or a localization prompt decision tree 3026,respectively. As illustrated, the diagnosis prompt decision tree 3022,the characterization prompt decision tree 3024, and/or the localizationprompt decision tree 3026 can each include a root node, a plurality ofinternal nodes that each branch from the root node or another internalnode, and a plurality of leaf nodes that each branch from the root nodeor an internal node. In particular, all of the fixed set of hierarchicaldiagnosis options, characterization options, and/or localization optionscan be represented by all of the leaf nodes of the diagnosis promptdecision tree, the characterization prompt decision tree, and/or thelocalization prompt decision tree, respectively. Each root node andinternal node can include any number of branches corresponding to aplurality of options that can be selected in response to a promptcorresponding to the node. Each node can be located in one of aplurality of levels of the corresponding prompt decision tree, whereeach level is characterized by a number of branches from the root node.

The plurality of prompt decision trees can be stored in the applicationdata as a data structures and/or abstract data type corresponding totrees in accordance with the structure of the corresponding decisiontrees. In some embodiments, other data structures and/or abstract datatypes are employed that indicate the plurality of decision trees and/orthat cause the labeling application to select a set of prompts that arepresented to the user as dictated by a corresponding prompt decisiontree, to select a corresponding set of options for each prompt asdictated by a corresponding prompt decision tree, and/or to select anordering of the set of prompts as dictated by a corresponding promptdecision tree, based on each user selection from the selected set ofoptions. This can include filtering a total plurality of prompts and/ora total plurality of options based on user selections, in accordancewith the corresponding decision tree. Thus, the labeling application canutilize a prompt selection algorithm that presents the prompts andoptions in accordance with at least one corresponding decision tree,even if the application data does not include any tree data structures.The examples presented in FIGS. 12C, 12D, and 12E present the behaviorexhibited in execution of the labeling application by illustrating theprompt selection algorithm for selecting for the set of prompts that arepresented to the user, the corresponding set of options for each prompt,and the ordering of the set of prompts. While the organization of datathat includes these prompts and/or options can be stored in acorresponding tree structure in the application labeling data, any otherdata structure can be employed to organize storage of these promptsand/or options. Furthermore, any other data structure can be employed toorganize storage of the instructions for such a prompt selectionalgorithm for selecting the prompts, the ordering of the prompts, and/oroptions associated with the prompts as discussed herein.

In presenting these prompt selection algorithms, FIGS. 12C-12Eillustrate the distinction between different internal nodes and leafnodes in accordance with the unique path from the root prompt. Forexample, diagnosis prompt 1.2 corresponds to the internal node reachedafter selecting selection option 1 from a set of selection options 1-Nfrom the diagnosis root prompt to reach characterization prompt 1, andthen selecting selection option 1.2 from a set of selection options1.1-1.M. As presented in FIGS. 12C-12E, M, N, and R correspond to anyinteger number of branches from the corresponding node. The use ofellipses “[ . . . ]” as presented in FIGS. 12C-12E indicates that anynumber of additional selections were made to reach the correspondingnode, and that the corresponding node can be at any level of the promptdecision tree. For example, leaf node 1.M.[ . . . ] indicates a leafnode that extends any number of branches from prompt 1.M. The promptdecision trees described herein can be of the same or differentconfigurations presented in FIGS. 12C-12E.

The labeling application can require that a leaf node is reached in eachprompt decision tree presented to the user, for example, where theinteractive interface will not advance to a next prompt decision treeand/or will not exit until the user continues to make selections toultimately advance to a leaf node. This ensures that the user'sannotation of the medical fully characterizes at least one abnormalityof the medical scan, while still producing structured, consumablelabeling data.

Some or all of nodes of one or more prompt decision trees can correspondto a prompt presented to a user via the interactive interface, where aselected branch from the node to a next node away from the root node isdetermined based on user input corresponding to selection of an optioncorresponding to the branch. Furthermore, some or all nodes of one ormore prompt decision trees can correspond to options presented to thelabeling application internally, where selected branches from thesenodes can be automatically determined for example, based on classifierdata 420 of the scan itself such as the modality of the scan and/oranatomical region of the scan; based on leaf nodes reached in of otherprompt decision trees; and/or based on another automatic determinationthat does not correspond to user input to the interactive interface.This can be utilized to advance from the root node to an internal nodeautomatically, where a user is presented with the plurality of promptsof a prompt decision tree starting from this automatically selectedinternal node. This can also be utilized to automatically advance to aninternal node or leaf node based on automatic determinationscorresponding to selections of branches to advance down the decisiontree, where the automatic determinations are made by the labelingapplication without user input.

The labeling application can include a plurality of diagnosis promptdecision trees, a plurality of characterization prompt decision trees,and a plurality of localization prompt decision trees for each modality,each anatomical regions, and/or for other scan classifier data 420. Forexample head CTs, chest x-rays, and chest CTs can each correspond totheir own diagnosis prompt decision tree, a characterization promptdecision tree, and localization prompt decision tree. In suchembodiments, the labeling application can automatically determine amodality, anatomical region, and/or other category of the medical scan,and can further automatically determine which of the plurality of treeswill be used based on the determined category.

The interactive interface can display each of a plurality of prompts toa user of the client device, one at a time, in accordance with thediagnosis prompt decision tree, characterization prompt decision tree,and/or localization prompt decision tree of the labeling applicationdata 3020. The user can select one of a fixed set of options indicatedby each of the plurality of prompts of the diagnosis prompt decisiontree as user input via interaction with the interactive interface 3075.The selected one of the plurality of options can dictate the next one ofthe plurality of prompts, in accordance with the corresponding promptdecision tree. These prompts can continue to be displayed in sequence tothe user, progressing to the next prompt indicated by the promptdecision tree as the user selects one of the plurality of optionspresented in accordance with each prompt, until a leaf node of theprompt decision tree is reached. The leaf node can indicate some or allof the selections made by the user in previous nodes of the promptdecision tree from a root node to the leaf node.

Furthermore, each set of options presented for a prompt corresponding toan internal node can include only an appropriate set of options that arepossible given the previous selections corresponding to branches fromthe root node to the diagnosis prompt node. This effectively filters theset of options presented to the user given the selections made so far,ensuring the set of presented options includes only valid options. Theseappropriate sets of options can be predetermined by an administrator orother user responsible for creating the prompt decision trees. In someembodiments, the medical scan hierarchical labeling system 3002automatically determines the options presented for each prompt of adecision tree based on a known set of rules, for example, correspondingto possible options given modality, anatomical regions, and userselections that must be made to reach the corresponding node. In someembodiments, the medical scan hierarchical labeling system 3002automatically optimizes an ordering of some or all of the prompts. Insome embodiments, identical prompts are included in multiple paths of aprompt decision tree, but are presented in different orders. In someembodiments, an ordering of prompts to each leaf node is determined tooptimize an expected number of branches necessary to reach a leaf node,optimize the average number of branches to reach a leaf node, to reduceor otherwise optimize the number of selection options presented in oneor more prompts of one or more nodes, and/or to otherwise optimize theprompt decision tree. In some embodiments, the set of prompts to reacheach leaf node is automatically determined to ensure the correspondingabnormality of each leaf node will be fully described in reaching theleaf node. In some embodiments, the number of this set of prompts isminimized, while still ensuring the corresponding abnormality of eachleaf node will be described to a necessary extent in reaching the leafnode.

The interactive interface can first present the user with a promptasking whether or not an abnormality is present, as shown in FIG. 12C.If the user selects “no”, the diagnosis prompt decision tree canimmediately branch to a leaf node indicating no abnormality is present,and the labeling data can correspond to a structured entry of themedical scan database indicating no abnormality is present. If the userselects the option indicating an abnormality is present, the interactiveinterface can display a diagnosis root prompt with a plurality ofselection options 1-N. The user can select a single one of the Nselection options to progress to the corresponding next diagnosis prompt1-N. For example, the user can select option 1, and the interactiveinterface will display diagnosis prompt 1 in response, indicating a setof selection options 1.1-1.M. If the user selects option 1.1.1 inresponse to diagnosis prompt 1.1, leaf node 1.1.1 is reached. Labelingdata corresponding to the diagnosis can indicate the diagnosiscorresponding to leaf node 1.1.1. The leaf node can effectively indicateall of the selections made from the root node to reach the leaf node.Thus, the labeling data generated in response to selection of the leafnode can further indicate each of the individual selections made by theuser to reach the leaf node, for example, where the labeling dataindicates selection option 1, selection option 1.1, and selection option1.1.1.

Each selection option 1, 1.1, and 1.1.1 can correspond to its own fieldof the relational database, and/or the relational database can includefields corresponding to only the set of leaf nodes, for example, with abinary indication of whether or not the diagnosis corresponding to eachleaf node is present. In some embodiments, the database includes asingle diagnosis field populated by one of the fixed set of optionsindicated by the one of the plurality of leaf nodes of the diagnosisprompt tree. The medical scan database can employ any structure with adiscrete set of possible diagnosis entries corresponding to acorresponding discrete set of diagnosis options corresponding to thediscrete set of leaf nodes of the diagnosis decision tree.

In some embodiments, the labeling application can automaticallydetermine a starting node of the diagnosis decision tree, correspondingto one of the interior nodes rather than the root node. In particular,the labeling application can automatically determine the starting nodeof the diagnosis decision tree based a determined modality of themedical scan, based on a determined anatomical region of the medicalscan, and/or other scan classifier data 420 of the medical scan. Leafnodes branching from the selected starting node will only includediagnosis options corresponding to abnormalities that can be detected inthe corresponding anatomical region of the medical scan and/or that canbe detected in the corresponding modality of the medical scan.

For example, the labeling application automatically proceed to selectionoption 1 that indicates plurality of abnormalities associated with thehead in response to determining that the medical scan is a head CT scan.In such embodiments, diagnosis prompt 1 will only present optionscorresponding to abnormalities that correspond to the head, and that canbe detected in a head CT. Any leaf nodes 1. [ . . . ] that branch fromdiagnosis prompt 1 will only correspond to final diagnosis thatcorrespond to the head, and that can be detected in a head CT. Forexample, the fixed set of options indicated by leaf nodes 1. [ . . . ]can include “brain tumor”, and “brain bleed”, but will not include“wrist fracture” or “pulmonary embolism.”

As another example, the labeling application can proceed automaticallyto a selection option 2.1 in response to determining that the medicalscan is a chest x-ray. For example, a selection option from thediagnosis root prompt can correspond to selecting the anatomical regionof the medical scan from a set of anatomical regions, where selectionoption 2 corresponds to the chest. Diagnosis prompt 2 can correspond toselecting the modality from a plurality of modalities, where selectionoption 2.1 corresponds to an x-ray. In such embodiments, diagnosisprompt 2.1 will only present options corresponding to abnormalities thatcorrespond to the chest, and that can be detected in an x-ray of thechest. Any leaf nodes 2.1.[ . . . ] that branch from diagnosis prompt2.1 will only correspond to final diagnosis that correspond to thechest, and can be detected in a chest x-ray. For example, the fixed setof options indicated by leaf nodes 2.1.[ . . . ] can include “ribfracture” or “pneumonia” because they can be identified by a chestx-ray. However, “pulmonary embolism” is not included in the leaf nodes2.1.[ . . . ], even though this condition is associated with the chest,because leaf nodes 2.1.[ . . . ] correspond to an x-ray modality, andbecause reviewing an x-ray alone is not sufficient to determine whethera pulmonary embolism is present or absent. However, if selection option2.2 corresponds to a CT scan, any leaf nodes 2.2. [ . . . ] that branchfrom diagnosis prompt 2.2 will only correspond to a final diagnosis thatcorresponds to the chest and that can be detected in a chest CT. Thus,leaf nodes 2.2.[ . . . ] can include “pulmonary embolism” as a selectionoption because this condition can be determined to be present or absentin reviewing a chest CT.

In some embodiments, a medical scan entry can include any variablenumber of differential diagnosis, where each of the variable number ofdifferential diagnosis are included in the same field or the variablenumber of different fields, and where each of the variable number ofdifferential diagnosis corresponds to a different one of the discreteset of leaf nodes of the diagnosis decision tree. In such embodiments,after reaching a leaf node of the diagnosis decision tree, the user canbe prompted with a question of whether or not there is anotherabnormality to characterize. In response to the user selecting “yes,”the interactive interface can return to the root prompt, and/or thestarting node determined based on the anatomical region and/or modality.Furthermore, a deeper starting node, branching from the initial startingnode but further away from the root node, can be selected automaticallyas a result of selection of the first one or more leaf nodes that havealready been selected, if these first one or more leaf nodes can narrowthe set of options possible for additional differential diagnoses. Insome embodiments, characterization and/or localization of theabnormality indicated in the first leaf node is completed in accordancewith the characterization prompt decision tree and/or the localizationprompt decision tree first, and once this characterization and/orlocalization is completed, the user is presented the option to identify,characterize, and/or localize additional abnormalities as differentialdiagnoses.

Once a leaf node is reached in the diagnosis decision tree and/or onceit is indicated that no further abnormalities are present, theinteractive interface can advance to prompts of the characterizationprompt decision tree. FIG. 12D illustrates an example embodiment of acharacterization prompt decision tree. The interactive interface canbegin with the root node of the characterization prompt decision treeand/or the labeling application can automatically advance to an internalnode based on the anatomical region and/or modality as discussed inconjunction with the diagnosis prompt decision tree Furthermore, as theset of options that can be used to characterize an abnormality dependson the type of abnormality itself, the labeling application canautomatically advance to an even deeper internal node based on the leafnode of the diagnosis decision tree that was ultimately selected. Forexample, as shown in FIG. 12C, a characterization prompt X progressingfrom leaf node 1.1.1 can correspond to any characterization prompt nodeof the characterization prompt decision tree determined to be anautomatic starting node given a diagnosis corresponding to leaf node1.1.1. Other leaf nodes of the diagnosis prompt decision tree can alsocause the interactive interface to automatically progress to their ownpre-determined starting node of the characterization prompt decisiontree.

From the selected starting node of the characterization prompt decisiontree, the corresponding diagnosis can be characterized based onselections from the starting node, where the interactive interfacepresents successive prompts corresponding to the nodes branching basedon selections to previous prompts, until a leaf node of thecharacterization prompt decision tree is reached. In embodiments wheremultiple leaf nodes for multiple abnormalities have already beenselected from the diagnosis prompt decision tree, each of theseabnormalities can be characterized separately, each with their ownautomatically determined starting node based on the corresponding leafnode, where a separate leaf node of the characterization decision treeis ultimately reached for each of these differential diagnoses.

The user can indicate if one or more differential diagnoses, identifiedvia selection of multiple leaf nodes of the diagnosis prompt decisiontree, correspond to side effects of a main one of the differentialdiagnoses. This can be performed in the plurality of characterizationprompts, where one or more side effects are indicated and characterizedas part of characterizing the main diagnosis. As one example,illustrated in FIG. 12D, once a diagnosis is characterized and a leafnode is reached, the user will be prompted with a question askingwhether or not the diagnosis is further characterized by any sideeffects. If so, a selected starting diagnosis prompt Y of the diagnosisprompt decision tree will be presented, and the interactive interfacewill present prompts of the diagnosis prompt decision tree until a leafnode of the diagnosis prompt decision tree is ultimately selected foreach side effect. Furthermore, each side effect can be further describedvia selection of leaf nodes in the characterization prompt decision treeand/or the localization prompt decision tree for the each side effect.As another example, after the user identifies a plurality ofdifferential diagnoses, the interactive interface can present a promptto select at least one main diagnosis from the plurality of differentialdiagnoses, and can further present a prompt to identify at least one ofthe remaining differential diagnoses as side effects of the maindiagnosis.

For example, if a head CT includes a brain tumor, brain bleed, andfracture, the brain bleed and fracture can be indicated as side effectsof the brain tumor. The user can reach leaf nodes of the diagnosisdecision tree corresponding to each of “brain tumor”, “brain bleed”, and“fracture”. In particular, after the “brain tumor” leaf node is reachedin the diagnosis prompt decision tree, the user can be prompted toidentify one or more side effects of the brain tumor as part ofcharacterizing the brain tumor via the characterization prompt decisiontree. In response to the user selecting to identify a side effect,prompts of the diagnosis prompt decision tree will be displayed untilthe leaf node corresponding to “brain bleed” is ultimately reached. Inresponse to the user selecting to identify an additional side effect,prompts of the diagnosis prompt decision tree will be displayed untilthe leaf node corresponding to “fracture” is ultimately reached. Inresponse to the user selecting that no more side effects are present,the user can continue characterizing and/or advance to localizing thebrain tumor itself. The brain bleed and fracture can also each becharacterized and localized by ultimately reaching leaf nodes of thecharacterization prompt decision tree and localization prompt decisiontree for each of these side effects.

Once a leaf node is reached in the characterization decision tree and/oronce the user indicates that no further side effects are present, theinteractive interface can advance to prompts of the localization promptdecision tree 3026, presented in FIG. 12E. The interactive interface canbegin with the root node of the localization prompt decision tree and/orthe labeling application can automatically advance to an internal nodebased on the anatomical region and/or modality as discussed inconjunction with the diagnosis prompt decision tree and thecharacterization prompt decision tree. Furthermore, as the set ofoptions that can be used to localize an abnormality can depend on thetype of abnormality itself, the labeling application can automaticallyadvance to an even deeper internal node based on the leaf node of thediagnosis prompt decision tree that was ultimately selected, and/orbased on the leaf node of the characterization prompt decision tree thatwas ultimately selected. For example, as shown in FIG. 12D, alocalization prompt Z progressing from leaf node 1.2.R can correspond toany localization prompt node of the localization prompt decision treedetermined to be an automatic starting node given a diagnosis and/orcharacterization corresponding to leaf node 1.2.R. Other leaf nodes ofthe characterization prompt decision tree can also cause the interactiveinterface to automatically progress to their own pre-determined startingnode of the localization prompt decision tree.

From the selected starting node of the localization prompt decisiontree, the corresponding abnormality can be localized based on selectionsfrom the starting node, where the interactive interface presentssuccessive prompts corresponding to the nodes branching based onselections to previous prompts, until a leaf node of the localizationprompt decision tree is reached. In embodiments where multiple leafnodes for multiple abnormalities have already been selected from thediagnosis prompt decision tree as differential diagnoses and/or sideeffects, each of these abnormalities can be localized separately, eachwith their own automatically determined starting node based on thecorresponding leaf node, where a separate leaf node of the localizationdecision tree is ultimately reached for each of these abnormalities.

In some embodiments, as an abnormality might be located in multipleplaces, the user can select any number of leaf nodes to localize eachabnormality. For example, as shown in FIG. 12E, after reaching a leafnode such as leaf node 1.1.1, the user can be presented with an optionto identify other locations in which the abnormality is present. Inresponse to the user selecting “yes,” the interactive interface canreturn to the root prompt, and/or the starting node determined based onthe anatomical region and/or modality. Furthermore, a deeper startingnode, branching from the initial starting node but further away from theroot node, can be selected automatically as a result of selection of thefirst one or more leaf nodes of the localization prompt decision treethat have already been selected, if these first one or more leaf nodescan narrow the set of options possible for additional locationsdiagnoses, such as narrowing to a set of adjacent locations or locationswithin proximity of locations selected thus far.

At least one internal node of the localization prompt decision tree canrequire that multiple leaf nodes are reached to characterize theabnormality. For example, as shown in FIG. 12G, localization prompt 1indicates that a number Q leaf nodes are required. Once a leaf node,such as leaf node 1.2.1 is reached, if the required number Q leaf nodeshave not yet been selected, localization prompt 1 will be presented andthe user can progress to leaf nodes, returning to localization prompt 1until the required number of leaf node selections Q have been made.While the diagnosis prompt decision tree of FIG. 12C nor thecharacterization prompt decision tree of FIG. 12D illustrate internalnodes or root nodes requiring multiple leaf nodes be reached, such nodesrequiring multiple leaf node selections can similarly be included in thediagnosis prompt decision tree or the characterization prompt decisiontree in other embodiments.

In some embodiments, this required number of leaf nodes can inherentlybe built into the localization prompt decision tree with single requiredselection from each node. For example, instead of returning tolocalization prompt 1 from leaf node 1.2.1, the leaf node 1.2.1 caninstead be an internal node 1.2.1 presenting localization prompt 1, withthe same set of M selection options this internal node, allowing theuser to continue to eventually reach the Q leaf node selections as aresult of progressing down deeper sets of branches and identifying eachof the Q leaf nodes as selections to the final, Qth leaf node along theway.

As a particular example, if a leaf node corresponding to “brain tumor”is selected from the diagnosis prompt decision tree, at least one lobeand at least one compartment must be selected in the step of localizingthe brain tumor. The localization prompt decision tree can present theuser with a plurality of lobe selection options and a plurality ofcompartment selection options, and can require that at least one leafnode corresponding to a lobe is selected, and that at least one leafnode corresponding to a compartment is selected. For example, once alobe is selected, the same prompt can be presented, excluding the lobethat has already been selected, until one of the plurality ofcompartment options is also selected. As another example, thelocalization prompt decision tree can present a first prompt with only aplurality of lobe options in an internal node, and once one of theplurality of lobe options is selected, can branch to the next node thatpresents a second prompt with only a plurality of compartment options,where the leaf node ultimately reached indicates the selected lobe andthe selected compartment. In such embodiments, each of the plurality oflobe selection options can branch to the identical prompt nodespresenting the plurality of compartment options.

In some embodiments, based on the diagnosis leaf node, characterizationleaf node, and/or localization leaf node, an additional set of questionsmay be necessary. For example, when leaf node N. [ . . . ] is reached,at least one additional prompt will be presented. This additional set ofquestions can correspond to prompts of at least one additional promptdecision tree, where the questions are hierarchical and dependent onprevious questions, and/or where leaf nodes of each additional promptdecision tree must be reached. The additional prompt set can be the sameor different for different leaf nodes. The additional prompt set candepend only on the diagnosis leaf node, characterization leaf node,localization leaf node, and/or on a combination of two or more nodes.The additional prompt set can be further based on differential diagnosesor side effects. The additional prompt set can be based on the modalityof the medical scan and/or the anatomical region of the medical scan.For example, if the medical scan corresponds to a head CT, theadditional question set includes ventricular system questions that willautomatically be presented via the interactive interface in response toan automatic determination that the medical scan is a head CT. In someembodiments, the additional prompt set is inherently included asadditional internal nodes of the diagnosis prompt decision tree,characterization prompt decision tree, and/or localization promptdecision tree, ultimately reaching a final leaf node that includesanswers to the additional questions. The answers to the additionalquestions can be included in the fixed format of the labeling data,where each of the additional questions similarly presents a fixed set ofoptions. Alternatively or in addition, at least one additional questioncan correspond to unstructured data, such as text or voice input by theuser, drawings and/or or shapes outlining one or more abnormalitiessuperimposed upon the medical scan as user input entered by the user,measurement data indicating size, shape, diameter, and/or volume ofabnormalities as identified by the user, a report entered by the user,or other unstructured data. In some embodiments, this unstructured datacan be mapped to the medical scan in the database, but can be separatefrom the rest structured labeling data.

In some embodiments, the medical scan includes a plurality of imageslices. The user can be prompted to select a proper subset of slicesand/or a single slice that includes an abnormality to be described inthe labeling data. A user selection that indicates a selected subset ofthe plurality of images slices of the medical scan can be received viauser input to the interactive interface. This prompt can be included inthe localization prompt decision tree and can dictate further prompts ofthe localization prompt decision tree based on a narrowed anatomicallocation corresponding to the proper subset of slices and/or the singleslice. Alternatively, this prompt can be presented separately fromprompts of the localization prompt decision tree, and the startinglocalization prompt can be selected based on the selected subset of theplurality of image slices. For example, if a subset of slices selectedby the user indicates the frontal lobe of a head CT, localizationprompts presented by the user interface will include optionscorresponding to the selection of the frontal lobe.

The user can be prompted to provide an urgency ranking as part of aprompt decision tree, additional question set, and/or as a final promptpresented to the user. The urgency ranking prompt can similarly includea fixed set of urgency ranking options for selection by the user toindicate an urgency associated with the diagnosis, associated withfurther review of the medical scan, and/or associated with furtherscans, tests, or appointments with the patients that may be necessary.The urgency ranking can be included in the labeling data, and/or can beutilized in triaging of the medical scan by one or more subsystems.

In some embodiments, one or more of the nodes of one or more of theprompt decision trees can be optional, where a user selection is notrequired. Such nodes can include a “skip” branch, indicating thatselection of one or more of the selection options is not necessary forthis prompt. Selecting the skip option can correspond to a branch thatadvances to a next node in the prompt decision tree.

While the discussion thus far indicates that the user first selects aleaf node of the diagnosis prompt decision tree, then a leaf node of thecharacterization prompt decision tree, and finally a leaf node of thelocalization prompt decision tree, prompts of the diagnosis promptdecision tree, the characterization prompt decision tree, and thelocalization prompt decision tree can be presented in any order. Forexample, the user might first localize the abnormality, and then basedon this localization, a starting node of the diagnosis prompt decisiontree is determined based on types of abnormalities that can exist and/orbe observed at the particular location indicated in the localizationleaf node.

Furthermore, while FIGS. 12C-12E present the diagnosis prompt decisiontree, characterization prompt decision tree, and localization promptdecision tree separately, prompts corresponding to diagnosis,characterization, and localization of one or more abnormalities can beincluded in a single prompt decision tree of the labeling applicationdata, where reaching a leaf node of the single prompt decision treeincludes selecting at least one diagnosis option from at least oneinternal node corresponding to a diagnosis prompt, selecting at leastone characterization option from at least one internal nodecorresponding to a characterization prompt, and selecting at least onelocalization option from at least one internal node corresponding to alocalization prompt. In some embodiments, the flow of prompts to atleast one leaf node includes a plurality of diagnosis prompts,characterization prompts, and localization prompts in any order, forexample, where first a localization prompt is presented, then adiagnosis prompt, then another localization prompt, and then acharacterization prompt. In some embodiments, multiple prompt decisiontrees that includes this mix of diagnosis prompts, characterizationprompts, and/or localization prompts are included in the labelingapplication data. Each of these multiple prompt decision trees cancorrespond to a different modality, a different anatomical region, adifferent modality/anatomical region pair, and/or other different scanclassifier data 420.

The labeling application can be utilized by multiple client devicescorresponding to multiple users, and each user can label multiplemedical scans by utilizing the labeling application. Labeling datagenerated over time by one or more users for one or more medical scanscan tracked in a user database 344, where the number of scans labeled byeach user of each client device is tracked. The user database canfurther track how many scans have been labeled for each of a set of scancategories corresponding to different modalities, different anatomicalregions, and/or other categories indicated by scan classifiers data 420.Users can be incentivized and/or rewarded for reaching a thresholdnumber of labeled scans in one or more scan categories and/or reaching atotal number of labeled scans. Similarly, users can be can beincentivized and/or rewarded for reaching and/or maintaining a thresholdlabeling rate in one or more scan categories and/or reaching and/ormaintaining a threshold labeling rate overall. In some embodiments,users can be incentivized to label scans in categories with a countand/or labeling rate that is below a threshold for that user,encouraging the user to expand their skills and label scans ofmodalities and/or anatomical regions they do not typically label. Insome embodiments, users can be incentivized to label scans in categorieswith a count and/or labeling rate that is below a global thresholdacross all users, for example, corresponding to scan types that are notaddressed enough by users across the system, to encourage the user tomeet this need. The incentives can include financial incentives, caninclude a favorable adjustment to performance score data and/orqualification data, and/or can include advancing a user to an expertstatus in one or more scan categories in which a number and/or labelingrate compares favorably.

In some embodiments, a medical scan can be automatically pre-processedto partition the medical scan in accordance with multiple anatomicalregions included within the medical scan. For example, a full body scancan be partitioned into a set of medical scan portions, where eachmedical scan portion corresponds to each of a set of anatomical regions.For example, the full body scan can be partitioned into medical scanportions corresponding to the head, chest, arm, leg, etc. for individuallabeling. These partitions can each be labeled by the same user or bydifferent users. For example, the medical scan hierarchical labelingsystem 3002 can perform this pre-processing step prior to transmissionof the medical scan to a client device. The different medical scanportions can be sent to different users for labeling based ondetermining each user has favorable qualification data and/orperformance score data for the corresponding anatomical region. Thelabeling data can be retrieved from all of the users and can be compiledfor the original medical scan to be mapped to the medical scan database.In some embodiments, the pre-processing step is performed after amedical scan is retrieved by a client device as part of execution of thelabeling application. Each partition can be presented in conjunctionwith prompt decision trees corresponding to the anatomical region ofeach partition and/or in conjunction with starting nodes of the promptdecision trees determined based on the anatomical region of eachpartition.

As presented in FIGS. 12A and 12B, the labeling application is executedby the client device, allowing the client device to generate all of thelabeling data locally, and this final labeling data is then transmittedback to the medical scan hierarchical labeling system 3002 via thenetwork. In some embodiments, some or all of the steps performed by theclient device in accordance with execution with the labeling applicationcan be instead executed by the medical scan hierarchical labeling system3002. This can be accomplished via additional transmissions between themedical scan hierarchical labeling system 3002 and the client device.

For example, some or all of the user input, such as user selection of aselection option of a prompt, can be transmitted via the network themedical scan hierarchical labeling system 3002. The medical scanhierarchical labeling system 3002 can utilize the corresponding promptdecision tree, stored in memory of the medical scan hierarchicallabeling system 3002, to determine the next prompt that will bepresented to the user and the corresponding set of options. This nextprompt and corresponding set of options can be transmitted to the clientdevice for display via the interactive interface, where one of this setof options is selected by the user via user input, and is transmittedback to the medical scan hierarchical labeling system 3002. Another nextprompt and another next set of options is determined by medical scanhierarchical labeling system 3002 based on the prompt decision tree fortransmission back to the client device. This process can continue untila leaf node is ultimately selected.

As another example, the medical scan hierarchical labeling system 3002can automatically select the starting node for one or more of the promptdecision trees based on the anatomical region, modality, and/or otherfeatures of the medical scan, in conjunction with transmission of themedical scan. An indicator of the starting node can be transmitted tothe client device, and the client device can present the plurality ofprompts beginning with the starting node based on the indicator of thestarting node received from the medical scan hierarchical labelingsystem 3002.

As another example, the medical scan hierarchical labeling system 3002can automatically select the starting node for one or more of the promptdecision trees based on the anatomical region, modality, and/or otherfeatures of the medical scan that is transmitted to the client device. Asubset of the corresponding one or more prompt decision trees isselected by the medical scan hierarchical labeling system 3002, wherethe root node of the subset is the selected starting node, and where thesubset includes internal nodes and leaf nodes that extend from thestarting node. This subset of the one or more prompt decision trees cantransmitted to the client device for use, where the remainder of the oneor more prompt decision trees is not transmitted to the client deviceand/or not stored by the client device. This subset of the one or moreprompt decision trees can be utilized by the client device to presentthe plurality of prompts.

As shown in FIG. 12F, a medical scan training set can be retrieved fromthe medical scan database 342. The medical scan hierarchical labelingsystem 3002, or another subsystem 101, can perform a training step 3090,for example, by utilizing the medical scan image analysis system andperforming training step 1352 of FIG. 7A. The medical scan training setcan include medical scans that were labeled by one or more clientdevices by utilizing the medical scan hierarchical labeling system 3002as discussed in conjunction with FIGS. 12A-12E. In particular, labelingdata mapped to the medical scans of the medical scan training set wasgenerated by client devices based leaf nodes reached in the diagnosisprompt decision tree, characterization prompt decision tree, and/orlocalization prompt decision tree. This labeling data is indicated foreach medial scan in the medical scan training set, for example, as astructured entry in the medical scan database. The labeling data foreach medical scan in the training set can be utilized as an outputfeature vector and/or output nodes of a neural network in the trainingstep 3090, where the output feature vector and/or output nodes arestructured in accordance with the fixed structure of the labeling data.The input feature vector and/or input nodes of the neural network cancorrespond to image data of each medical scan in the training set,patient history data of each medical scan in the training set, and/orother data of the corresponding medical scan entry of each medical scanin the training set. Once the model is trained, model data can betransmitted to the medical scan function analysis database 346 to beutilized by one or more other subsystems 101. Alternatively or inaddition, the model data can be transmitted to the medical picturearchive integration system 2600 to be utilized in performance ofinference functions by the medical picture archive integration system2600.

The model data can be utilized by the medical scan hierarchical labelingsystem 3002, or another subsystem 101, as shown in FIG. 12G to generateinference labeling data for one or more new medical scans to be labeled,for example, retrieved from the medical scan database 342, by performingan inference function 3095 that utilizes the trained model. For example,the medical scan image analysis system 112 and/or the inference step1354, detection step 1372, and/or abnormality classification step 1374of FIG. 7B can be utilized to generate inference labeling data for newmedical scans. The inference labeling data can correspond to the samefixed structure of the labeling data generated by utilizing the labelingapplication, dictated by the fixed set of abnormality options, the fixedset of classification options, and the fixed set of localizationoptions. The inference labeling data can indicate probability values forone or more of the fixed set of abnormality options, one or more of thefixed set of classification options, and/or one or more of the fixed setof localization options. These probability values can indicateprobabilities that one or more of the fixed set of abnormality optionsis present in the scan and/or in a region of the scan corresponding to aprobability matrix corresponding to the probability value, that one ormore of the fixed set of classification options describes the one ormore of the fixed set of abnormality options, and/or that theabnormality is located in the one or more of the fixed set oflocalization options. The inference labeling data can be mapped to thenew medical scan in the medical scan database and/or can be transmittedto a client device for display to a user via a user interface.

In various embodiments, medical scan hierarchical labeling systemincludes a medical scan database that stores a plurality of medical scanentries, at least one processor, and a memory. The memory storeslabeling application data that includes application operationalinstructions and a plurality of prompt decision trees. The plurality ofprompt decision trees includes a diagnosis prompt decision tree, acharacterization prompt decision tree, and a localization promptdecision tree. Each of the plurality of prompt decision trees includes aroot node, a set of internal nodes, and a set of leaf nodes. Each rootnode and each of the set of internal nodes correspond to one of aplurality of prompts. Each root node and each of the set of internalnodes include a set of branches that each correspond to one of adiscrete set of selection options for the one of the plurality ofprompts. The memory further stores a medical scan relational databasethat stores a plurality of medical scan entries. The medical scanrelational database includes a discrete set of fields corresponding tothe leaf nodes of the plurality of prompt decision trees.

The executable instructions, when executed by the at least oneprocessor, cause the medical scan hierarchical labeling system totransmit, via a network, labeling application data to a client devicefor storage. The application operational instructions of the labelingapplication data, when executed by at least one client device processorof the client device, cause the client device to execute a labelingapplication. The executable instructions further cause the medical scanhierarchical labeling system to transmit via the network, a medical scanto the client device.

Execution of the labeling application by the client device causes theclient device to, in response to receiving the medical scan, display,via an interactive interface presented on a display device associatedwith the client device for display to a user associated with the clientdevice, image data of the medical scan. The client device automaticallydetermines a starting diagnosis prompt by selecting one of the set ofinternal nodes of the diagnosis prompt decision tree based on ananatomical region of the medical scan and further based on a modality ofthe medical scan. The client device displays, via the interactiveinterface, a plurality of diagnosis prompts of the diagnosis promptdecision tree, in succession, beginning with the starting diagnosisprompt, in accordance with corresponding nodes of the diagnosis promptdecision tree until a first one of the set of leaf nodes of thediagnosis prompt decision tree is ultimately selected. The interactiveinterface progresses to each next one of the plurality of diagnosisprompts by selecting one of the set of branches of each correspondingnode in accordance with each of a plurality of corresponding userdiagnosis selections, received via user input. Each of the plurality ofcorresponding user diagnosis selections corresponds to one of thediscrete set of selection options for each one of the plurality ofdiagnosis prompts displayed via the interactive interface.

The client device automatically determines a starting characterizationprompt by selecting one of the set of internal nodes of thecharacterization prompt decision tree based on the anatomical region ofthe medical scan, based on the modality of the medical scan, and furtherbased on the first one of the set of leaf nodes of the diagnosis promptdecision tree. The client device displays, via the interactiveinterface, a plurality of characterization prompts, in succession,beginning with the starting characterization prompt, in accordance withcorresponding nodes of the characterization prompt decision tree until afirst one of the set of leaf nodes of the characterization promptdecision tree is ultimately selected. The interactive interfaceprogresses to each next one of the plurality of characterization promptsby selecting one of the set of branches of each corresponding node inaccordance with each of a plurality of corresponding usercharacterization selections, received via user input. Each of theplurality of corresponding user characterization selections correspondsto one of the discrete set of selection options for each one of theplurality of characterization prompts displayed via the interactiveinterface.

The client device automatically determines a starting localizationprompt by selecting one of the set of internal nodes of the localizationprompt decision tree based on the anatomical region of the medical scan,and further based on the modality of the medical scan. The client devicedisplays via the interactive interface, a plurality of localizationprompts, in succession, beginning with the starting localization prompt,in accordance with corresponding nodes of the localization promptdecision tree until a first one of the set of leaf nodes of thelocalization prompt decision tree is ultimately selected. Theinteractive interface progresses to each next one of the plurality oflocalization prompts by selecting one of the set of branches of eachcorresponding node in accordance with each of a plurality ofcorresponding user localization selections, received via user input.Each of the plurality of corresponding user localization selectionscorresponds to one of the discrete set of selection options for each oneof the plurality of localization prompts displayed via the interactiveinterface.

The client device transmits, via the network, labeling data thatincludes a set of labels indicating the first one of the set of leafnodes of the diagnosis prompt decision tree, the first one of the set ofleaf nodes of the characterization prompt decision tree, and the firstone of the set of leaf nodes of the localization prompt decision tree.

The executable instructions, when executed by the at least one processorof the medical scan hierarchical labeling system, further cause themedical scan hierarchical labeling system to receive, via the network,the set of labels from the client device, and to populate a medical scanentry of the medical scan in the medical scan relational database basedon the set of labels.

FIGS. 13A-13E presents an embodiment of a medical scan labeling qualityassurance system 3004. The medical scan labeling quality assurancesystem 3004 can be utilized to routinely gauge how a set of usersresponsible for labeling medical scans are performing. This can beaccomplished by sending a quality assurance set of medical scans toclient devices of the set of users, where each user generates labelingdata for each of the medical scans in the set. The quality assurance setof medical scans is also sent to a user identified as an expert forlabeling. The set of labeling data generated by the expert user,corresponding to labeling data for each medical scan in the set, can beconsidered a set of golden labeling data. The golden labeling data inthis set can be mapped to the corresponding medical scans in the medicalscan database. The set of medical scans and corresponding set of goldenlabeling data can be utilized as training data to train one or moremedical scan analysis functions. The set of golden labeling data can besent to the set of users for review. The set of golden labeling data canbe utilized to generate performance score data for the set of labelingdata generated by each user. The performance score data can be mapped tothe user in the user database and can be utilized to determine whetherto dismiss one or more users and/or can be utilized to advance one ormore users to an expert status.

As shown in FIGS. 13A-13E, medical scan labeling quality assurancesystem 3004 can communicate bi-directionally, via network 150, with themedical scan database 342, user database 344, and/or other databases ofthe database storage system 140, with one or more client devices 120,and/or, while not shown in FIG. 12A, one or more subsystems 101 ofFIG. 1. In some embodiments, the medical scan labeling quality assurancesystem 3004 is an additional subsystem 101 of the medical scanprocessing system 100, implemented by utilizing the subsystem memorydevice 245, subsystem processing device 235, and/or subsystem networkinterface 265 of FIG. 2A. In some embodiments, the medical scan labelingquality assurance system 3004 is implemented by utilizing, or otherwisecommunicates with, the central server 2640. For example, some or all ofthe databases of the database storage system 140 are populated withde-identified data generated by the medical picture archive integrationsystem 2600. In some embodiments, the medical scan labeling qualityassurance system 3004 can receive de-identified medical scans,annotation data, and/or reports directly from the medical picturearchive integration system 2600. For example, the medical scan labelingquality assurance system 3004 can request de-identified medical scans,annotation data, and/or reports that match requested criteria, forexample, corresponding to training set criteria. In some embodiments,some or all of the medical scan labeling quality assurance system 3004is implemented by utilizing other subsystems 101 and/or is operable toperform functions or other operations described in conjunction with oneor more other subsystems 101. In some embodiments, the medical scanlabeling quality assurance system 3004 is integrated within and/orutilizes the medical scan assisted review system 102 and/or the medicalscan annotator system 106. In some embodiments, the medical scanlabeling quality assurance system 3004 is integrated within and/orutilizes the medical scan hierarchical labeling system 3002.

In some embodiments, the medical scan labeling quality assurance system3004 is utilized in conjunction with one or more other subsystemsanother subsystem 101 responsible for sending and/or triaging medicalscans to users for labeling, and mapping the generated labeling data tothe medical scans in the medical scan database. The normal operation ofthis overarching system can include triaging or otherwise assigning eachof a plurality of medical scans to one of a set of users in the systemfor labeling. Thus, labeling data generated for each these medical scanis generated by only one user that was assigned to the medical scan.This labeling data can be mapped to the medical scans in the medial scandatabase and/or can be used as training data for to train a computervision model and/or to train one or more medical scan analysisfunctions. This labeling data generated in normal operation of thesystem may not be reviewed by other users in the system, unless the useris later flagged as poor performing in conjunction with the qualityassurance process.

On the contrary, the set of medical scans selected in conjunction withthe quality assurance process are sent to multiple users that generatelabeling data, such as all of the users in the system. Labeling datawill be generated for this set of medical scans by all of the users forcomparison to the golden labeling data generated for these medical scansby the expert user. This quality assurance set of medical scans cancomprise only a small subset of the plurality of medical scans reviewedby the set of users. For example, a user may label 100 medical scans aweek, where 10 of these medical scans are sent every week in conjunctionwith the quality assurance process. Labeling data generated by the forthese 10 medical scans of the quality assurance set are compared to thegolden labeling data to generate performance score data as describedherein. Labeling data for the remaining 90 of the medical scans ismapped to the medical scan in the medical scan database normally,without review. Sets of medical scans sent to the set of users within atimeframe corresponding to the schedule of the quality assurance processeach include the same quality assurance set of medical scans, as well asadditional medical scans that will be processed normally. Thus, theintersection of these sets of medical scans sent to the set of userswithin the time frame includes only the quality assurance set of medicalscans.

In some embodiments, the set of medical scans of the quality assuranceset are indistinguishable by users from the remaining medicals scansthey receive in the timeframe. This can ensure that users are labelingscans of the quality assurance set normally, and/or can ensure that thelabeling data of the quality assurance set is a reflection of normallabeling behavior of the user. Alternatively, the set of medical scansof the quality assurance set can always be the first medical scans sentto the set of users in the timeframe. In some embodiments, the remainingmedical scans for normal processing are only sent to the user if theperformance score data for the labeling data of the quality assuranceset compares favorably to the threshold. This can ensure that users withpoor performance are not allowed to generate their own labeling data formedical scans in the medical scan database.

As shown in FIG. 13A, the medical scan labeling quality assurance system3004 can initiate performance of the quality assurance process bygenerating a request for a set of medical scans for transmission themedical scan database, and by generating request for a set of clientdevice identifiers for a set of users for transmission the userdatabase. The medical scan labeling quality assurance system 3004 canreceive a set of medical scans from the medical scan database inresponse and can receive a set of client device identifiers from theuser database. As illustrated in the example presented in FIGS. 13A-13E,the set of medical scans of the quality assurance process received fromthe medical scan database includes M medical scans, and the set ofclient device identifiers corresponding to a set of N users. The Mmedical scans corresponds to the quality assurance set of medical scans.

As the medical scan labeling quality assurance system 3004 routinelyperforms a quality assurance process to assess the performance of usersat regular intervals, the request for the set of medical scans andrequest for set of client device identifiers can be transmitted inresponse to determining a fixed amount of time has elapsed since a lasttime that the quality assurance process was performed and/or bydetermining the current time compares favorably to a scheduled time toconduct the quality assurance process. This step can be performed toinitiate each subsequent quality assurance process at the routine timeintervals and/or in accordance with the subsequently scheduled qualityassurance processes.

The medical scan set request can indicate particular identifiers orcriteria that the set of medical scans should meet. In some embodiments,a set of medical scans that meet the criteria are randomly orpseudo-randomly selected. In some embodiments, the set of medical scansdo not have any diagnosis data 440 or other labeling data associatedwith them in the medical scan database and/or are otherwise selected inresponse to determining they need to be labeled. The criteria used toselect the set of medical scans can include a number of medical scans tobe selected, at least one desired modality, at least one desiredanatomical region, other features of scan classifier data 420, anurgency level, a recency that the medical scans were added to thesystem, or other criteria. This criteria can be determined via userinput by an administrator and/or can be determined automatically by themedical scan labeling quality assurance system 3004, for example, basedon one or more scan categories determined to best assess the set oflabelers. For example, in response to determining previous sets ofmedical scans did not include a threshold number of medical scanscorresponding to a scan category, at least one medical scan, or at leasta threshold number of medical scans, of that scan category can beincluded in the set of medical scans.

In some embodiments, types of scans that users perform poorly on can beselected to assess whether users are improving with time and/or toassess whether users should be dismissed from labeling these types ofscans in the future. For example, in response to determining one or moreof the set of users have performance score data indicating poorperformance for a scan category and/or in response to determining thepoorest performing scan category across all of the users in the set ofusers, at least one medical scan, or at least a threshold number ofmedical scans, of this that scan category can be included in the set ofmedical scans.

Alternatively, the criteria can be selected to include types of scansthat users perform well on can be selected to ensure users are stillperforming strongly with time and/or to assess whether users should beassigned an expert status for these types of scans. For example, inresponse to determining one or more of the set of users have performancescore data indicating strong performance for a scan category and/or inresponse to determining the strongest performing scan category acrossall of the users in the set of users, at least one medical scan, or atleast a threshold number of medical scans, of this that scan categorycan be included in the set of medical scans.

A set of client device identifiers for a selected set of users can alsobe determined, for example, by fetching the identifiers from the userdatabase. The set of users can correspond to users that regularlygenerate labeling data for medical scans to populate entries in themedical scan database and/or to generate labeling data utilized astraining data. For example, the set of users can correspond to allusers, or a subset of users, that utilize the labeling application togenerate labeling data that is assigned to medical scans in the medicalscan database in accordance with the medical scan hierarchical labelingsystem. The same or different set of users can be selected for eachroutine quality assurance process over time, based on the same ordifferent criteria determined over time.

In some embodiments, a subset of the set of users is selected based ontheir corresponding performance history, for example, indicated by theirperformance score data 530 in the user database 344. In someembodiments, the set of users can include only to users with performancescore data that falls within a determined performance score range. Forexample, the high end of the performance score range can correspond to aperformance score threshold indicating that users meeting or exceedingthis threshold need not be assessed at regular intervals. In someembodiments, this high end of the performance score range corresponds toa performance score threshold indicating that users meeting or exceedingthis threshold are assigned an expert status. In some embodiments,determining whether performance score data compares favorably to thehigh end includes determining that the user has labeled at least athreshold number of medical scans and/or at least a threshold proportionof medical scans with a high end performance score. The low end of theperformance score range can correspond to a performance score thresholdindicating that users failing to meet this threshold are no longerallowed to label medical scans at all, are on probation for a durationof time, are on probation until they can pass a more rigorous test oftheir labeling capabilities, have been dismissed from the systementirely, or otherwise are not currently labeling any scans for thegiven timeframe. In some embodiments, the performance score range onlyhas a high end. In some embodiments, the performance score range onlyhas a low end.

The performance score range can be set for one or more scan categoriesof medical scans in the quality assurance set, and can be the same ordifferent for each category. The medical scan labeling quality assurancesystem 3004 can select only users that fall within all of theperformance score ranges for their corresponding categorized performancescores. Alternatively the medical scan labeling quality assurance system3004 can select users that fall within at least one of the performancescore ranges for their corresponding categorized performance scores.

Performing the quality assurance process can include selecting differentquality assurance sets of medical scans for different selected sets ofusers based on different criteria. For example, a first set of users canbe sent only chest x-rays, and a second set of users can be sent headCTs. This grouping of users can be based on determining the types ofscans the users specialize in, the types of scans the users labels mostfrequently, the types of scans the users perform most poorly on, thetypes of scans the users label least frequently, or other criteria. Insome embodiments, some users are sent multiple of these sets of medicalscans, where one user is included in the first set and the second setand is thus sent the chest x-ray quality assurance set as well as thehead CT quality assurance set. Each quality assurance set can havegolden labeling data generated by the same or different expert user. Forexample, users identified as experts in each scan category can be sentthe corresponding set of medical scans to generate the corresponding setof golden labeling data.

The quality assurance process can be part of an onboarding process fornew users to the system, where these new users are not assigned scansfor normal labeling until they pass the onboarding process. In suchembodiments, these users can receive only the set of quality assurancescans in each timeframe, until their performance score data comparesfavorably to an onboarding threshold and/or until threshold number oftimeframes for which the user participated in the quality assuranceprocess exceeds a threshold. In some embodiments, poor performing usersare automatically pushed back to a beginner state and must complete theonboarding process again before continuing to label scans normally.

Once the medical scan set and the set of client devices have beendetermined for the quality assurance process for the current timeframe,the set of medical scans 1-M can be sent to client devices for each ofthe set of users 1-N as illustrated in FIG. 13B. The client devices 1-Ncan be implemented utilizing client device 120. The medical scans 1-Mcan be displayed via interactive interface 3175. For example, the usercan generate labeling data for each medical scan 1-M, in sequence, oneat a time, in accordance with display of each medical scan 1-M insequence, one at a time, by the interactive interface 3175. The labelingapplication of the medical scan hierarchical labeling system 3002 can beutilized to enable the user to interact with the interactive interface3175 to enable each client device generate labeling data for eachmedical scan. In some embodiments, the labeling data is other annotationdata and/or unstructured data discussed herein, and interfaces of themedical scan assisted review system 102 and/or medical scan annotatorsystem 106 can be utilized to enable each client device to generate thisunstructured labeling data based on user input. For example, thelabeling data can include text entered by the user, can include regionof interest data or location data indicating the location ofabnormalities identified by the user, can indicate measurement dataindicating size, shape, diameter, and/or volume of abnormalities asidentified by the user, and/or can include other diagnosis data 440. Thelabeling data 1-M generated by each client device for the set of medicalscans is transmitted to the medical scan labeling quality assurancesystem 3004.

As illustrated in FIG. 13C, the medical scan labeling quality assurancesystem 3004 can request a client device identifier for an expert user,and can receive the expert client device identifier in response. Theexpert user can be selected randomly from a set of expert users, cancorrespond to a highest ranked user, and/or can otherwise be selected inresponse to determining the user has an expert status. For example, theexpert status can be indicated in the expert data 543 of the userprofile entry.

The expert user can be determined to be an expert for a modality,anatomical region and/or other scan classifier data 420 corresponding toat least one of the medical scans of the medical scan set. Thisdetermination can be based on categorized performance score data 534 ofthe user profile entry, categorized qualification data of the userprofile entry, and/or another indication that the user is an expert in acategory corresponding to the type of some or all of the medical scans.In some embodiments, a user identified as an expert in a plurality ofcategories corresponding to a set of categories characterizing all ofthe set of medical scans is selected, ensuring the expert user is highlyqualified to generate golden labeling data for all of the scan types inthe set of medical scans. In some embodiments, all of the medical scansin the set are of the same category, and a user identified as an expertof the corresponding category is selected. For example, when a pluralityof quality assurance processes corresponding to different scancategories are performed, an expert user can be identified for each ofthe plurality of quality assurance processes based on having expertisein the corresponding category.

The client device identifier for the expert user can be utilized to sendthe set of medical scans of the quality assurance process to expertclient device 3120. Expert client device 3120 can be implemented byutilizing a client device 120 that is associated with the expert user.The medical scans 1-M can be displayed via interactive interface 3177,which can be the same or different from interactive interface 3175. Forexample, the expert user can generate golden labeling data for eachmedical scan 1-M, in sequence, one at a time, in accordance with displayof each medical scan 1-M in sequence, one at a time, by the interactiveinterface 3177. The labeling application of the medical scanhierarchical labeling system 3002 can be utilized to enable the expertuser to interact with the interactive interface 3177 to enable theexpert client device generate golden labeling data for each medicalscan. In some embodiments, the golden labeling data is other annotationdata and/or unstructured data discussed herein, and interfaces of themedical scan assisted review system 102 and/or medical scan annotatorsystem 106 can be utilized to enable the expert client device togenerate this unstructured labeling data based on user input. The goldenlabeling data 1-M generated by the expert client device for the set ofmedical scans is transmitted to the medical scan labeling qualityassurance system 3004.

The medical scan labeling quality assurance system 3004 can also sendeach set of labeling data 1-N to the expert client device for review.The labeling data provided by some or all of the users for some or allmedical scans can be displayed to the expert user to assist the expertuser in creating the global data set. In some embodiments, the expertuser can elect to display labeling data from each of the N users one ata time for a single medical scan in conjunction with the display of thesingle medical scan. In some embodiments, the expert user can elect todisplay a subset or all of the labeling data from each of the N usersone at a time for a single medical scan in conjunction with the displayof the single medical scan. The labeling data from the N users can bedisplayed as text, can outline, highlight or otherwise indicateabnormalities identified in the medical scan by at least one of the Nusers, or can otherwise indicate the labeling data. If labeling data formultiple users are displayed simultaneously, labeling data of differentusers can be distinguished, for example, by displaying labeling data indifferent colors, displaying user identifiers in conjunction with eachof the labeling data, or otherwise distinguishing the labeling datacreated by different users. In some embodiments, historical performancedata and/or qualifications for a user can be displayed in conjunctionwith the labeling data for that user, allowing the expert user toevaluate contrasting opinions based on how well the corresponding usershave performed in the past and/or how qualified the corresponding usersare.

In some embodiments, a plurality of experts are identified. The set ofmedical scans are sent to a plurality of client devices corresponding tothe plurality of experts, and the golden labeling data is generated byperforming a consensus function on the labeling data received from eachof the plurality of experts, such as the annotation consensus functiondescribed in conjunction with FIG. 4040.

The medical scan labeling quality assurance system 3004 can pre-processthe labeling data sets 1-N before transmission to the expert clientdevice. The medical scan labeling quality assurance system 3004 canidentify the most popular labeling data and/or average labeling data foreach medical scan 1-M by evaluating the labeling data from each of the Nusers for each medical scan. For example, the medical scan labelingquality assurance system 3004 can generate consensus data as discussedin conjunction with the discussion of the medical scan annotator system106 in FIGS. 14A-14V. The medical scan labeling quality assurance system3004 can also identify labeling data from one or more user that differsmost from the popular labeling data and/or the consensus data. In someembodiments, the medical scan labeling quality assurance system canaggregate labeling data assigned to a medical scan by each user fordisplay to the expert as a histogram, indicating how many users agreedon each label for a medical scan. This information can be sent to theexpert client device for display in conjunction with each medical scanand/or in conjunction with display of raw labeling data from the Nusers, to further assist the expert in determining the golden labelingdata for each medical scan.

The interactive interface 3177 can further allow the expert user toenter correction data for the labeling data submitted by the N users.The correction data can be entered individually for the labeling data ofeach of the N users, and can be entered individually for each of the Mmedical scans for each of the N users, where a correction data set for auser corresponds to the correction data for a single user's labelingdata for all of the medical scans 1-M. For example, the expert user canenter text in a text box presented by the interactive interfaceindicating errors in the labeling data for one of the users, allowingthe expert user to identify reasons that the labeling data was corrector incorrect, and/or to identify reasons the labeling data entered bythe user is be associated with common pitfalls. The expert user caninteract with interactive interface 3177 to draw on or otherwisesuperimpose shapes, lines, and text over the medical scan itself, forexample, to indicate abnormalities the user missed and/or to indicatebenign features that were improperly identified as abnormalities. Thecorrection data can indicate any other problems with the labeling dataentered by the user, can indicate feedback to the user that generatedthe labeling data, and/or can include positive and/or encouragingcomments or other feedback indicating the user generated completelycorrect or mostly correct labeling data. The correction data can furtherinclude recommendations for areas that the user should study for furtherreview, for example, indicating fields of medicine, types of medicalscans, types of prompts of one or more prompt decision trees, types ofanatomical regions and/or types of abnormalities corresponding to weakperformance and/or corresponding to areas where the user could use morepractice. In this fashion, the correction data can be intended as alearning tool for the user to improve in the future. In someembodiments, aggregate correction data is generated based on the expertuser's evaluation of a user's performance overall over the medical scans1-M. This can be determined based on common pitfalls or errorsidentified by the expert user based on the set of labeling data for theset of medical scans viewed as a whole, and/or based on strengths and/ortrends that were consistently performed well by the user over the set ofmedical scans as a whole.

Each set of correction data 1-N can also indicate scoring dataidentified by the expert user. This can include overall scoring data,scoring data for each of the set of labeling data for a user, scoringpoints awarded for strong performance, and/or scoring points deductedfor errors identified by the expert in the labeling data. The expertuser can identify a ranking of the N users' performance for each medicalscan or for review over all medical scans, and/or each set of correctiondata can indicate the ranking identified by the expert relative to theother N−1 users.

As shown in FIG. 13D, the medical scan labeling quality assurance system3004 can receive the golden labeling data for each of the M medicalscans, and can receive correction data sets 1-N indicating correctiondata sets for each user. A scoring step 3150 can be performed todetermine performance score data for each of the labeling data sets 1-N.The performance score data for each of the N users can be generatedautomatically by the medical scan labeling quality assurance system 3004based on comparing the set of golden labeling data to the set oflabeling data for each of the N users.

The performance score data generated for a user can be an aggregatescore for labeling data of all of the medical scans in the set. Anindividual performance score can be calculated by comparing labelingdata for each of the M medical scans for a user to the correspondinggolden labeling data of the set of golden labeling data. The performancescore data for a user can be an aggregate score based on these Mindividual performance scores. This can include an average score of theM individual performance scores, can include a median score of the Mindividual performance scores, can include a maximum score of the Mindividual performance scores, can include a minimum score of the Mindividual performance scores, and/or can include some other overallscore based on the M individual performance scores. In some embodiments,the performance score data preserves each of the M individualperformance scores.

More favorable scores can be generated for labeling data that is thesame as, or deviates only slightly from, the corresponding goldenlabeling data. Less favorable scores can be generated for labeling datathat is different from, or deviates greatly from, the correspondinggolden labeling data. For example, a difference between the labelingdata and corresponding golden labeling data for a single medical scancan be calculated. This can include calculating the Euclidian distancebetween a feature vector of corresponding to the user's labeling datafor a medical scan and the feature vector of the corresponding goldenlabeling data, where a higher performance score is assigned to a userwhose labeling data is a smaller Euclidian distance from thecorresponding golden labeling data, and a lower performance score isassigned to a user whose labeling data is a larger Euclidian distancefrom the golden labeling data.

Alternatively or in addition, the performance score for an individualmedical scan can be presented as a binary indicator, indicating whetheror not the labeling data was correct based on a comparison to the goldenlabeling data. The binary indicator can be determined based on absolutecorrectness. Alternatively, the binary indicator or can indicate whetheror not a difference of the labeling data from the golden labeling datawas small enough to compare favorably to a correctness threshold, wherethe binary indicator indicates whether or not the labeling data was“mostly correct” in accordance with the correctness threshold.

The performance score data can further be generated based on correctlabeling of each of a plurality of differential diagnoses indicated ingolden labeling data for a medial scan. For example, if golden labelingdata for one of the medical scans indicates X abnormalities, acorresponding performance score for each user for this medical scan canbe generated based on a number or proportion of the X abnormalities thatwere correctly identified.

The performance score data can further be generated based on the set ofcorrection data corresponding to each set of labeling data. For example,the scoring data and/or rankings determined by the expert upon review ofthe sets of labeling data can be utilized in generating the performancescore data. The performance score data can be set equal to the scoringdata and/or rankings set by the expert user. Alternatively, generatingthe performance score data for a user can include weighing the scoresand/or ranks assigned by the expert user with a first weight, andweighing the automatically generated scoring data based on thecalculated difference between the set of labeling data of the user andthe set of golden labeling data with a second weight. The performancescore data is calculated in accordance with summing the product of thefirst weight and the expert determined scoring data with the product ofthe second weight and the automatically calculated scoring data.

Alternatively or in addition, the performance score data can further begenerated based on the user's performance relative to other users and/orrelative to average performance. Consider an example where most usersincorrectly labeled a first medical scan, but a first user labeled thefirst medical scan correctly. This can warrant a more favorableperformance score for the first example than a second example where allor almost all users, including the first user, correctly labeled asecond medical scan. The first medical scan is determined to be a morechallenging medical scan to label correctly than the second medicalscan, which is determined to be easy to label correctly. Thus, the firstuser is rewarded more heavily for correctly labeling the first,challenging medical scan than the second, easy medical scan. Theperformance score data for the first user includes a first performancescore for labeling data of the first medical scan that is more favorablethan a second performance score for labeling data of the second medicalscan, even when the labeling data for both the first medical scan andsecond medical scan are equally “correct” relative to the goldenlabeling data, to reward the first user more heavily for their correctlabeling of the more challenging medical scan.

Consider this same example with this first, challenging medical scanthat was labeled incorrectly by almost all users and this second, easymedical scan that was labeled incorrectly by almost all users. A seconduser in the set of users was among the users that incorrectly labeledthe first medical scan, and was also the only user to incorrectly labelthe second medical scan. Incorrectly labeling the easy medical scan canwarrant a more unfavorable performance than incorrectly labeling thechallenging medical scan. Thus, the second user is penalized moreheavily for incorrectly labeling the second, easy medical scan than thefirst, challenging medical scan. The performance score data for thesecond user includes a first performance score for labeling data of thefirst medical scan that is more favorable than a second performancescore for labeling data of the second medical scan, even when thelabeling data for both the first medical scan and second medical scanare equally “incorrect” relative to the golden labeling data, topenalize the second user more heavily for their incorrect labeling ofthe easy medical scan.

Following this mentality, some or all of the performance score data forindividual medical scans for each user can be generated based on howsimilar or dissimilar a user's labeling data is from the most popularlabeling data, from consensus labeling data, and/or from averagelabeling data. A histogram generated for labeling data received from theN users for each medical scan can also be utilized to determine howcommon or uncommon a user's labeling data is. A second differencebetween the labeling data of a medical scan and corresponding consensuslabeling data, corresponding average labeling data, and/or correspondingmost common labeling data can be calculated. This can includecalculating the Euclidian distance between a feature vector ofcorresponding to the user's labeling data for a medical scan and thefeature vector of the corresponding consensus labeling data,corresponding average labeling data, and/or corresponding most commonlabeling data.

This second calculated difference can be utilized in conjunction withthe first calculated difference from the golden labeling data, and/orcan be utilized in conjunction with a binary indicator indicatingwhether or not the labeling data was correct based on a comparison tothe golden labeling data, and/or indicating whether or not a differenceof the labeling data from the golden labeling data was small enough tocompare favorably to a correctness threshold. As a specific example,consider a neutral score of 0. The performance score can be generated byassigning a score of 1 to correct (or mostly correct) labeling data andby assigning a score of −1 to incorrect (or mostly incorrect) labelingdata for a single medical scan. A performance score for the singlemedical scan can be calculated multiplying D, denoting the magnitude ofthe calculated second difference from average, consensus, or most commondata, by the assigned score of 1 or −1 based on whether this labelingdata was correct or incorrect, resulting in D or −D. The performancescore data over all of the medical scans M for a single user can becomputed by calculating this single performance score for the user'slabeling data for each medical scan, and summing the M calculatedperformance scores.

Some or all of the correction data discussed herein can be generatedautomatically by the medical scan labeling quality assurance systemand/or the client device locally in conjunction with executingapplication data received from the medical scan labeling qualityassurance system that causes the client device to generate thecorrection data automatically. For example, the performance score datagenerated for each user can be utilized to generate the set ofcorrection data for each user, and/or a comparison of the labeling datato the golden labeling data for each medical scan of a user can beutilized to generate correction data for each medical scan for eachuser. Some or all of the automatically generated correction data can bedisplayed to the expert client device for review as suggested correctiondata, and the expert client device can utilize this suggested correctiondata in generating the final correction via the interactive interface3177, for example, by providing edits to and/or additions to thesuggested correction data.

Once the performance score data is generated for each user, each of theN performance score data is sent to a client device of a respective oneof the N users via the network, as shown in FIG. 13D. Each user can viewtheir respective performance score data indicating their overallperformance and/or individual scores 1-M via the interactive interface3175. For example, the interactive interface can present the set ofmedical scans 1-M and the respective individual performance scores tothe user. In some embodiments, ranking information indicating how wellthe user performed relative to the other N−1 users is also transmittedto the user, allowing the user to gauge their relative performance foreach medical scan individually and/or to gauge their overall relativeperformance. Furthermore, all of the performance score data can be sentto all of the client devices allowing the user to see how much theirindividual performance scores for each medical scan and/or allperformance scores deviate from those of the other N−1 users.

Each set of correction data generated via expert user input and/orgenerated automatically can also be transmitted to a client device of arespective one of the N users via the network, as shown in FIG. 13D. Theset of correction data can be displayed to the user in conjunction withthe user's set of labeling data and/or the set of medical scans. Forexample, each medical scan can be displayed one at a time to the user insequence, along with the correction data and/or the user's originallabeling data. This can include displaying the golden labeling data inconjunction with the user's labeling data, for example, where the user'slabeling data is distinguishable from the golden labeling data and/orwhere differences between the user's labeling data from the goldenlabeling data is visually indicated. This can include displaying anytext and/or superimposed drawings of the correction data in conjunctionwith each medical scan.

In some embodiments, data corresponding to other users can be sent tosome or all additional users of the set of users 1-N. A client device ofa first user can receive sets of labeling data generated by other users,performance score data for other users, and/or set of correction datagenerated for other users. In some embodiments, all of the sets oflabeling data 1-N, all of the performance score data 1-N, and/or all ofthe set of correction data 1-N are sent to all of the users 1-N. Theinteractive interface 3175 can display the labeling data, performancescore data, and/or correction data of other users in the same fashionthat they can view their own labeling data, performance score data,and/or correction data, for example, where this information is displayedin conjunction with display of each medical scan one at a time. In someembodiments, the user can view their own labeling data and labeling dataof one or more other users, for example, in the same fashion that theinteractive interface 3177 displays labeling data for some or all of theusers at the same time, where different labeling data of different usersis distinguishable. In some embodiments, the histogram data, consensuslabeling data, average labeling data, common labeling data, and/or orother aggregate analysis of all of the sets of labeling data 1-N can beviewed by the user, allowing the user to recognize how similar ordissimilar their labeling data is from that of other users.

In some embodiments, the expert user can indicate custom teachingexamples for a user based on weaknesses of the user in the correctiondata for one or more of the M medical scans. The expert user canidentify labeling data of at least one different user for the one ormore M medical scans as some or all of the custom teaching example dataincluded in the correction data. The medical scan labeling qualityassurance system 3004 can send only labeling data identified as teachingexamples to the corresponding user based on the indication in thecorrection data. The medical scan labeling system can also send thecorresponding individual performance scores and/or the correspondingcorrection data for the labeling data of the indicated custom teachingexamples.

The expert user can also indicate global teaching examples that would beappropriate for any user. For example, the expert user can identifylabeling data for at least one medical scan that succumbs to a commonpitfall and/or that avoids a common pitfall. As another example, theexpert user can identify labeling data corresponding to a strongperformance that serves as a good example of proper labeling and/orcorresponding to a weak performance that serves as an example of poorlabeling that should be avoided. The labeling data identified as theglobal teaching data can be sent to all of the client devices 1-N fordisplay via the interactive interface. In some embodiments, thecorresponding correction data and/or individual performance scores ofthe global teaching examples can also be transmitted to all of theclient devices 1-N for display via the interactive interface. In someembodiments, the labeling data, corresponding correction data, and/orcorresponding performance score data of the global teaching can be madeavailable to additional users in the system, and can be transmitted toadditional corresponding client devices. The labeling data,corresponding correction data, and/or corresponding performance scoredata can be stored as historical global teaching examples and can befetched or sent automatically to additional users. For example, thelabeling data, corresponding correction data, and/or correspondingperformance score data can be mapped to the corresponding medical scanin the medical scan database, and/or can be stored in other memory ofthe medical scan labeling quality assurance system 3004 and/or inanother database of the database storage system 140.

The medical scan quality assurance labeling system can automaticallydetermine some or all of the custom teaching examples by identifyingcustom subsets of labeling data of other users based on the performancescore data, where the custom subsets are determined to be mostappropriate based on the user's errors, based on the types of medicalscans the user struggled with, and/or based on the types ofabnormalities the user struggled to identify properly. Only the customsubsets of labeling data of other users are be transmitted to thecorresponding user for display via the interactive interface. Thecorresponding correction data and/or individual performance scores ofthe custom subsets of labeling data can also be transmitted to thecorresponding user for display via the interactive interface. In someembodiments, the custom subsets are identified in the historical globalteaching data and/or other historical labeling data from a previousquality assurance process.

The medical scan labeling quality assurance system 3004 can alsoautomatically determine some or all of the global teaching examples sentto users and/or stored in a database as historical data. For example themedical scan labeling quality assurance system 3004 can automaticallydetermining common pitfalls and selecting labeling data that succumbs tothe common pitfall and/or that avoids the common pitfall. For example,the common pitfalls can be identified based on the common labeling dataof the histogram data that compares unfavorably to the global labelingdata and/or is otherwise determined to be incorrect. Common pitfalls canalso be identified by the expert and can be received from the expertclient device, and can be utilized to identify common pitfalls in thesets of labeling data. and can be further identified based ondetermining common labeling data that As another example, the medicalscan labeling quality assurance system 3004 can select labeling datacorresponding to highest scoring labeling data and/or that correspondsto lowest scoring labeling data for one or more medical scan categoriesand/or one or more abnormality types as global teaching examples.

FIG. 13E illustrates transmitting the golden labeling data 1-M to themedical scan database. For example, golden labeling data for the Mmedical scans and/or golden labeling data for other medical scansgenerated by experts in other performances of the quality assuranceprocess can be utilized to generate a golden training set to train amedical scan analysis function, for example as discussed in conjunctionwith FIG. 12F. As another example, the medical scan image analysissystem 112 can utilize the golden labeling data as output labels for aneural network in training the model as discussed in FIG. 7A. Thetrained medical scan analysis function can be utilized by one or moresubsystems 101 and/or by the medical picture archive integration system2600 to generate diagnosis data and/or other inference data for newmedical scans.

FIG. 13E also illustrates transmitting the performance score data foreach of the users to the user database. A user profile entry of eachuser can be updated based on this received user profile data. Theperformance score data can correspond to accuracy data 531 of the user'sperformance score data 530, and/or can correspond to categorizedperformance score data 534 based on the corresponding category of themedical scan. The performance score data 530 of the user's user profileentry can be updated to reflect the most recent performance score data,can reflect an average of all of the performance score data generatedfor that user for all of their performance score data generated overtime, and/or can otherwise be updated based on the new performance scoredata received from the medical scan labeling quality assurance system3004. As discussed previously, as the performance score data is updatedfor a user over multiple quality assurance processes, the performancescore data of the user can dictate whether the user is identified as anexpert, whether the user is included in the set of users for futurequality assurance processes, and/or whether the user is put on probationand/or banned from generating labeling data for other medical scansreceived as part of the normal functioning of the system.

The sets of correction data 1-N can also be sent to the user database oranother database, for example, allowing each user to fetch their own setcorrection data in the future and/or allowing other users to fetchcorrection data for review. This can further allow historical sets ofcorrection data to be fetched for custom or global teaching examplesidentified by the expert user and/or identified by the medical scanlabeling quality assurance system 3004 automatically for transmissionsome or all of the N users and/or additional users. In some embodiments,each original set of labeling data is also stored in conjunction withthe corresponding set of correction data.

In various embodiments, a medical scan labeling quality assurance systemincludes a medical scan database that includes a plurality of medicalscans, a user database that includes a plurality of user profilescorresponding to a plurality of users of the medical scan labelingquality assurance system, a processing system that includes a processor,and a memory that stores executable instructions. The executableinstructions, when executed by the processing system, cause the medicalscan labeling quality assurance system to select a set of users from theuser database in response to determining a scheduled interval haselapsed. A set of medical scans are selected from the medical scandatabase. The set of medical scans are transmitted, via a network, to aset of client devices associated with the set of users. The set ofmedical scans are displayed to the set of users via a first interactiveinterface displayed by a set of display devices corresponding to the setof client devices. A set of labeling data are received from each of theset of client devices via the network. Each set of labeling data isgenerated by a corresponding one of the set of client devices, and eachset of labeling data includes labeling data for each of the set ofmedical scans. The labeling data for each of the set of medical scans isgenerated by the corresponding one of the set of client devices inresponse to at least one prompt to provide the labeling data via thefirst interactive interface in conjunction with display of the each ofthe set of medicals scan.

An expert user is selected from the user database. The expert user isnot included in the set of users. The set of medical scans istransmitted, via the network, to an expert client device associated withthe expert user. The set of medical scans are displayed to the expertuser via a second interactive interface displayed by an expert displaydevice corresponding to the expert user. A golden set of labeling datais received from the expert client device via the network. The goldenset of labeling data is generated by the expert client device andincludes golden labeling data for each of the set of medical scans. Thegolden labeling data for each of the set of medical scans is generatedby the expert client device in response to at least one prompt toprovide the golden labeling data via the second interactive interface inconjunction with display of the each of the set of medicals scans.

A set of performance score data is generated by generating performancescore data for each corresponding set of labeling data by comparing thelabeling data of each set of labeling data to the golden labeling dataof the set of golden labeling data. Each performance score data of theset of performance score data is assigned to a corresponding one of theset of users that generated the corresponding set of labeling data. Eachof a set of user profile entries is updated in the user database foreach corresponding one of the set of users based on the performancescore data of the set of performance score data assigned to thecorresponding one of the set of users. Each performance score data ofthe set of performance score data is transmitted to a corresponding oneof the set of client devices for display, via the first interactiveinterface, to a corresponding one of the set of users to which the eachperformance score data is assigned.

FIGS. 14A-14B present embodiments of medical scan annotator system 106,for example, when utilized in conjunction with the medical scanhierarchical labeling system 3002 and/or the medical scan labelingquality assurance system 3004. As illustrated in FIG. 14A, the medicalscan annotator system 106 can select a medical scan from the medicalscan database 342 for transmission via network 150 to one or more clientdevices 120 associated with a selected user set 4010 corresponding toone or more users in the user database 344. A medical scan can beselected for annotation based on an assigned priority and/or based on aturn-based queue, for example, based on the scan priority data 427 ofthe corresponding medical scan entry 352. The client device 120 of eachuser of the selected user set 4010 can display one or more receivedmedical scans to the via the interactive interface 275 displayed by adisplay device corresponding to the client device 120, for example, bydisplaying medical scan image data 410 in conjunction with the medicalscan assisted review system 102.

The interactive interface 275 displayed by client devices 120 of eachuser in the selected user set 4010 can include a prompt to provideannotation data 4020 corresponding to the medical scan. This can includea prompt to provide a text and/or voice description via a keyboardand/or microphone associated with the client device. This can alsoinclude a prompt to indicate one or more abnormalities in the medicalscan, for example, by clicking on or outlining a region corresponding toeach abnormality via a mouse and/or touchscreen. For example, theinteractive interface can prompt the user whether or not an abnormalityis present. If the user indicates an abnormality is present, theinteractive interface can prompt the user to identify the region thatincludes the abnormality. This can include allowing the user to scrollthrough one or more slices, to identify one or more slices that containthe abnormality, and to select a region of the one or more slices thatcontains the abnormality. Once the region is identified, the interactiveinterface can prompt the user to provide descriptive informationclassifying an abnormality based on its size, type, etc. To aid the userin providing this information, the user interface can automatically cropone or more slices based on the identified region and/or zoom in on theidentified region. In various embodiments, the medical scan can bepresented for annotation by utilizing the medical scan assisted reviewsystem 102, for example, presented in the new annotation mode. Theinteractive interface 275 can present the medical scan by utilizinginterface features indicated in the display parameter data 470 and/orthe interface preference data 560 of the user, and/or the user canindicate the annotation data via the interactive interface 275 byutilizing interface features indicated in the display parameter data 470and/or the interface preference data 560 of the user. For example, someor all of the annotation data 4020 can correspond to, or beautomatically generated based on, user input to the interactiveinterface.

Annotation data 4020 can be transmitted from each client device of usersin the selected user set 4010 to the medical scan annotator system 106,for example, in response to receiving input data via the interactiveinterface indicating that the annotations are complete. The annotationdata 4020 can be raw annotation data corresponding directly to the userinput, or can be further processed by the client device beforetransmission. For example, a more precise region corresponding to eachabnormality can be determined automatically based on the user input andby determining actual boundary points of the abnormality by utilizingimage processing techniques and/or text and/or voice input can beprocessed and/or parsed, for example, by utilizing a medical scannatural language analysis function and/or medical report analysisfunction to generate medical codes 447 or other diagnosis data 440corresponding to the medical scan. Such processing can also be performedby the medical scan annotation system 106 and/or another subsystem whenthe raw annotation data is received.

The medical scan annotator system 106 can evaluate the set annotationdata 4020 received from the selected user set 4010 to determine if aconsensus is reached, and/or generate a final consensus annotation 4030,for example, by performing an annotation consensus function 4040. Forexample, consider a selected user set 4010 that includes three users. Iftwo users annotate a medical scan as “normal” and the third userannotates the medical scan as “contains abnormality”, the annotationconsensus function 4040 performed by medical scan annotator system 106may determine that the final consensus annotation 4030 is “normal” byfollowing a majority rules strategy. Alternatively, the medical scanannotator system 106 can determine that a consensus is not reachedbecause one of the users indicated that an abnormality is present, andthat the medical scan should not be passed off as normal because a levelof confidence that the scan is normal, determined by a calculatedconsensus confidence score 4050, does not exceed a consensus confidencethreshold. The confidence thresholds required for consensus can differfor different types of scans and/or severity of diagnosis.

If the medical scan annotator system 106 determines that a consensus isachieved, it can automatically generate the final consensus annotation4030, and can map this final consensus annotation to the medical imagein the medical scan database in diagnosis data 440, and/or transmit theconsensus annotation to an originating entity of the medical scan. Themedical scan annotator system 106 can also map the calculated consensusconfidence score to the medical image in the confidence score data 460.In some embodiments, a truth flag 461 will automatically be assigned toall final consensus annotation 4030 in the confidence score data 460and/or will automatically be assigned to final consensus annotation 4030that exceeds a truth threshold. In some embodiments, annotation data4020 received from each user and/or a corresponding annotationconfidence score can also be stored in the medical database, mapped tothe corresponding user and/or the corresponding performance score in theannotation author data 450.

In some embodiments, for example where annotation data 4020 includesseveral attributes, the annotation consensus function 4040 performed bythe medical scan annotation system 106 can determine whether a consensusis reached by calculating a difference between two or more receivedannotation data 4020, for example, by generating a feature vector forannotation data 4020 received from each user. Each feature vector can begenerated based on keywords, medical codes, abnormality location in themedical scan, abnormality size and/or shape in the medical scan, aclassification of the abnormality, or other attributes listed inannotation data 4020 received from each user. Performing the annotationconsensus function 4040 can further include calculating the Euclidiandistance or other vector distance between the two or more featurevectors. Performing the annotation consensus function 4040 can furtherinclude determining if consensus is reached by determining if theaverage of these Euclidian distances is below a certain discrepancythreshold, for example, after determining and removing outlierannotations from the set. Similarly, the annotation consensus function4040 can further include determining if consensus is reached by firstgenerating the final consensus annotation 4030, and then calculating theEuclidian distance between each annotation feature vector and the finalconsensus annotation 4030, where consensus is determined to reached andthe final consensus annotation is confirmed only if the average of thesecalculated Euclidian distances is below a certain discrepancy threshold.The annotation consensus function 4040 can calculate the final consensusannotation 4030 itself by creating a consensus feature vector, whereeach attribute of the consensus feature vector is determined bycalculating a mean, median or mode of each corresponding annotationfeature extracted from all of the received annotation data 4020. In thisfashion, calculating the consensus confidence score 4050 can includecalculating such an average Euclidian distance, where distances withlarger magnitudes correspond to lower or otherwise less favorableconsensus confidence scores 4050, and where distances with smallermagnitudes correspond to higher or otherwise more favorable consensusconfidence scores 4050. Alternatively or in addition, the finalconsensus annotation 4030 can be generated based on the most closelymatching annotations and/or based on another average, for example,calculating an average identified region that includes an abnormality.

The annotation consensus function 4040 further determine whether or notconsensus is reached based on overall or categorized performance scoredata 530 and/or qualification data 540 of each user in the selected userset 4010. For example, each annotation data 4020 can be weighted basedthe performance scores and/or qualifications of the corresponding user.In the example where two users annotate a medical scan as “normal” and athird user annotates a medical scan as “contains abnormality”, themedical scan annotator system 106 may determine that the consensus is“contains abnormality” based on the third user having a much higherperformance score and/or being more highly qualified than the first twousers. The final consensus annotation 4030 can be generated based on theannotation received from a user with the highest ranking in the categorycorresponding to the medical scan. The final consensus annotation 4030can be generated based on calculating a weighted average annotation bycomputing a weighted consensus feature vector, where feature vectors ofhigher ranked users receive a higher weight. In some embodiments, eachfeature of the feature vector can be computed using a different set ofuser weights, for example, where the different feature weights for eachuser is determined based on corresponding category-based performancescore data and/or qualification data.

Alternatively or in addition, the performance score data associated withthe interface features of the interactive interface 275 used by eachuser to annotate the image can also be utilized to weight the differentannotations in reaching consensus. Such weights can be applied whengenerating a consensus feature vector, where each annotation featurevector is weighted according to the performance score data of one ormore corresponding interface features used by the corresponding user.

In some embodiments, confidence scores for each individual annotationcan also be calculated for each user's annotation, and the consensusconfidence score 4050 can be generated based on these confidence scores,for example, based on an average confidence score, based on confidencescores of annotation data that matches the final consensus annotation4030, etc. In some embodiments, the final consensus annotation 4030 canbe generated based on these confidence scores, for example, whereannotation feature vectors are weighted based on a correspondingconfidence score. The confidence scores for each annotation data 4020can be generated automatically, for example, based on performance scoredata 530 as discussed herein. Individual confidence scores and/or aconsensus confidence score 4050 can also be updated retroactively as newannotation data is received, for example, if new annotation data isreceived from another user, for example corresponding to an expertreview when consensus is not reached, and/or if new annotation data isautomatically generated by a subsystem after the consensus data isgenerated.

The medical scan annotator system 106 can also utilize auto-generatedannotation data of the medical scan to determine if consensus is reachedand/or to generate the final consensus annotation 4030. Theauto-generated annotation data can be automatically generated by medicalscan annotator system 106 by utilizing one or more medical scan analysisfunctions. The auto-generated annotation data can also be retrieved fromthe medical scan database 342 if it was generated by a subsystem 101previously. One or more auto-generated annotations can be assigned theirown weights and/or confidence scores, for example, based on the modelaccuracy data 631 and/or another determined performance of the functionand/or subsystem responsible for creating each auto-generatedannotation. Each auto-generated annotation data can be thus treated asan annotation from another user, and can be used to determine ifconsensus is reached and/or to generate the consensus annotation in thesame fashion.

Alternatively, the auto-generated annotation can be merely verifiedbased on the annotation data 4020 received from the selected user set4010 by determining that the user annotations are close enough to theauto-generated annotation based on the discrepancy threshold. Forexample, this process may be utilized by the medical scan diagnosingsystem 108 to perform the output quality assurance step. Theauto-generated annotation can be sent to the selected user set 4010 aspart of this verification process, for example, displayed by eachinteractive interface 275 in conjunction with the medical scan assistedreview system 102 as displayed annotation data, and the annotation data4020 received from the selected user set 4010 can be includeverification of and/or corrections of the auto-generated annotation.Alternatively, the medical scan can be sent without the auto-generatedannotation and/or the auto-generated annotation can be hidden from viewas part of a blind review, to ensure that the users are not biased increating annotation data by the auto-generated annotation.

FIG. 14B illustrates an embodiment of the medical scan annotator system106 upon determining that a consensus is not achieved, for example,because the calculated consensus confidence score 4050 does not exceedthe consensus confidence threshold. The medical scan annotator systemcan select an expert user, for example, a user whose qualification data540 indicates they are an expert in the category corresponding to themedical scan or who otherwise is identified as an expert based on theirperformance score data. The expert can receive the medical scan on acorresponding client device and annotate the image, for example, wherethe interactive interface 275 displays the medical scan image data 410in conjunction with the medical scan assisted review system 102, andwhere the interactive interface utilizes interface features indicated inthe display parameter data 470 of the medical scan and/or indicated inthe interface preference data 560 of the user profile entry 354 of theexpert user. The expert can view the annotation data 4020 generated bythe selected user set 4010, for example, presented as the displayedannotation data of the medical scan assisted review system 102.Annotation data 4020 of each user can be displayed one at a time and theexpert user can elect to advance to the next user's annotation data4020. Alternatively, all of the annotation data 4020 can be displayedsimultaneously for example, in different colors corresponding to eachuser's annotations and/or overlaid as translucent, highlighted regions,for example, where a portion of the highlighted region is more opaquewhen multiple users agree that the portion is included in theabnormality. In other embodiments, the annotation data 4020 can behidden from the expert user, and the expert user can enter their ownannotations in conjunction with a blind review to reduce bias.

Expert annotation data 4070 can be generated automatically, and can betransmitted automatically to the medical scan annotation system 106. Themedical scan annotator system can automatically assign the receivedexpert annotation data 4070 as the final consensus annotation 4030,and/or can assign a truth flag 461 to the expert annotation data 4070 inthe confidence score data 460 of the medical scan. Alternatively, theexpert annotation data 4070 can be compared to the previous annotationdata 4020 and determine if consensus has been reached. For example, theexpert annotation data 4070 and the annotation data 4020 can becollectively utilized by the annotation consensus function 4040, wherethe expert annotation data 4070 is assigned its own, higher weight thanthe other annotations. If consensus has still not been reached, themedical scan annotation system can continue to transmit the image otherusers and processing received annotations until consensus is reached,for example, selecting a new selected user set 4010 and/or selecting anew expert user.

The user profile entries 354 of each user in the selected user set 4010and/or each expert user can be automatically updated by the medical scanannotator system 106 or another subsystem 101 by generating and/orupdating performance score data 530 for each user based comparing theirannotation to the final consensus annotation 4030. For example, theaccuracy score data 531 of the performance score data 530 can begenerated by calculating the Euclidian distance between a feature vectorof a user's annotation and the feature vector of the consensusannotation as described previously, where a higher performance score isassigned to a user whose annotation is a smaller Euclidian distance fromthe consensus, and a lower performance score is assigned to a user whoseannotation is a larger Euclidian distance from the consensus. Theefficiency score data 532 of the performance score data can beautomatically generated, for example, based on an annotation durationdetermined based on a difference between a first time that each userreceived the medical scan and a second time each user completed theannotation. The efficiency score data 532 can be further based on adifference between the annotation duration of each user and an averageannotation duration computed for annotation durations of the selecteduser set. Aggregate performance data for each user can be generateand/or updated based on past accuracy and/or efficiency scores, based onhow many scans have been annotated in total, based on measuredimprovement of the user over time, etc. Similarly, the performance scoredata 630 corresponding to medical scan analysis functions utilized togenerate the auto-generated annotation data can be generated and/orupdated by comparing the auto-generated annotation data to the finalconsensus annotation 4030 in a similar fashion and/or by comparing thecomputed annotation duration of a corresponding medical scan analysisfunctions to other computed annotation durations of other medical scananalysis functions that generated auto-generated annotation data for themedical scan.

The selected user set 4010 can be selected based on the performancescore data 530 and/or qualification data 540 of each user correspondingto previous uses only the medical scan annotation system 106, orcorresponding to usage of several subsystems 101. For example, a medicalprofessional with a user profile indicating that he/she ranks above acertain threshold in annotating CT scans and/or indicating that he/sheis highly qualified in the study of the lungs can be automaticallyselected by the medical scan annotator system to annotate a triagedmedical scan identified as a lug CT scan. The size of the selected userset 4010 that receive a medical scan can be optimized based on thequality of the users selected, for example, based on calculating theprobability of reaching consensus and/or calculating the probabilitythat a consensus confidence score will be above a confidence threshold,and ensuring the probability falls above a probability threshold. Forexample, a first medical scan can be sent to a two medical professionalswith high scores, qualifications, rankings, or correct annotationpercentages. A second medical scan may be sent to ten medicalprofessionals with lower scores or qualifications based on calculatingthat the probability of a correct consensus probability falls above aprobability threshold.

In some embodiments, the medical scan annotator system 106 can firstselect a medical scan for annotation automatically, and in response, theselected user set 4010 can be determined automatically to annotate theselected medical scan based on determining users with highly rankedoverall scores and/or based on categorized performance data 534 and/orqualification data 540 that corresponds to an identified scan classifierdata 420 of the selected medical scan. Alternatively or in addition, theselected user set 4010 can be determined based on the size of a queue ofmedical scans already assigned to each user. For example, the selecteduser set 4010 can correspond to users with matching qualifications thatcorrespond to the scan classifier data 420 and/or correspond to userswith the lowest queues of other medical scans to annotate.

In other embodiments, the medical scan annotator system 106 can firstdetermine one or more available users automatically, for example, basedon medical scan queue lengths for each user in the system and/or inresponse to one or more users requesting to annotate a medical scan. Insuch cases, some or all of these identified users can be added to theselected user set 4010, and the medical scan can be selected based oncorresponding categorized performance data 534, qualification data 540or other relevant user profile data of users in the selected user set4010.

FIGS. 14C-14V present example embodiments of a user interface of amedical scan annotator system 106, for example, presented in conjunctionwith the medical scan assisted review system 102. Some or all featurespresented in FIGS. 14C-14V can also be utilized in conjunction withother subsystems and can be included in the interface features. FIGS.14C-14G present interface features for chest CT nodule characterization,and can be displayed in conjunction with a chest CT scan. Annotationdata 4020 can be generated based on user selections in the userinterface, and can be used to populate abnormality classification data445 for abnormality classifier categories 444 such as “nodulespiculation”, “nodule lobulation”, “nodule texture”, “nodulecalcification”, “nodule sphericity” and/or “nodule internal structure”for the associated medical scan. FIGS. 14H-14J present interfacefeatures for presentation to a user in conjunction with an identifyingchest CT nodule, allowing a user to add new contours for one or morescans for a patient, for example, over multiple years, and indicatemalignancy. As shown in FIG. 14K, the scan can be presented inconjunction with these interface features. FIGS. 14L-140 presentinterface features for presentation to a user in conjunction withidentifying abnormalities in a chest x-ray. Users can classify eachabnormality and draw a shape around each abnormality in the scan.

FIG. 14P presents a view of a chest x-ray presented via the interfacebefore a user identifies regions of interest, and FIG. 14Q presents aview of the chest x-ray via the interface after the user identifiesregions of interest of multiple abnormalities, indicated by sevenpolygons 1022. FIG. 14R presents interface features for comparing chestx-ray severity for multiple patients, displayed in conjunction withmultiple x-rays that can be displayed in adjacent views or can bedisplayed one at a time where the user can toggle between them. A usercan compare multiple scans corresponding to multiple patients, andprovide feedback indicating differences between the patients, comparingif one patient's case is more severe than another, or determine which oftwo scans appears to be more normal.

FIGS. 14S-14V present interface features for chest x-ray triageclassification, displayed in conjunction with a chest x-ray. A user canselect abnormality classification data that can be used to generateannotation data 4020 and/or to populate abnormality classification data445. As shown, some or all abnormality classification categoriesdisplayed, which can be determined based on abnormality classifiercategories 444, can be presented, and hierarchal subcategories can bepresented in response to a user selecting one of a plurality ofabnormality classification categories that are present.

In some embodiments, the medical scan hierarchical labeling system 3002is integrated within and/or utilizes features described in conjunctionwith the medical scan annotator system 106. In particular, theinterfaces presented in some or all of FIGS. 14C-14V can be utilized bythe medical scan hierarchical labeling system 3002, where optionspresented in in some or all of FIGS. 14C-14V can correspond to a set ofselection options of a node in accordance with a prompt decision treeand/or where prompt decision trees of the medical scan hierarchicallabeling system 3002 utilize at least one of the prompts and/or sets ofoptions presented in in some or all of FIGS. 14C-14V as one or moreprompts of one or more prompt decision trees, where the annotation data4040 corresponding to the labeling data generated by the client device.The fixed set of diagnosis, characterization, and/or localizationoptions can include some or all of the selections presented in in someor all of FIGS. 14C-14V, where fields of a medical scan entry correspondto some or all of the selections presented in in some or all of FIGS.14C-14V and/or have valid entries corresponding to some or all of theselections presented in some or all of FIGS. 14C-14V.

The interactive interface can present hierarchical sets of options as auser advances through a prompt decision tree as presented in FIGS.14S-14V, where each set of options is presented as an indented list inaccordance with advancing to a deeper layer of the prompt decision tree,and where each set of options is not presented until the correspondingselection is made. For a hierarchical decision tree corresponding to theprompts presented in FIGS. 14S-14V, labeling data generated in responseto a user selecting “submit and next” at FIG. 14V can indicate a leafnode of a first abnormality, with diagnosis, characterization, andlocalization data fully described as “pulmonary vasculature”,“plethora”, “diffuse”, “left”, and “lobe-left upper”, and can furtherindicate a leaf node of a second abnormality with diagnosis,characterization, and localization data fully described as“mediastinum”, “compression of structure”, and “inferior-anterior”.

In some embodiments, the medical scan labeling quality assurance system3004 is integrated within and/or utilizes features described inconjunction with the medical scan annotator system 106. In particular,the interactive interface 275 of the medical scan annotator system 106can correspond to the interactive interface 3075 and/or 3175. Theannotation data 4040 can correspond to labeling data generated by aclient device 120 in conjunction with execution of a labelingapplication in conjunction with the medical scan hierarchical labelingsystem 3002 and/or the medical scan labeling quality assurance system3004. The expert annotation data 4070 medical scan annotator system 106can correspond to the golden labeling data generated by expert clientdevice 3120. The medical scan labeling quality assurance system 3004 candetermine the difference between labeling data and golden labeling dataand/or can generate performance score data as described in performingthe annotation consensus function 4040 of the medical scan annotatorsystem 106.

FIG. 15A presents a flowchart illustrating a method for execution by amedical scan hierarchical labeling system 3002 that stores executionalinstructions that, when executed by at least one processor, cause themedical scan hierarchical labeling system 3002 to perform the stepsbelow.

Step 5002 includes transmitting, via a network, labeling applicationdata to a client device for storage. The labeling data includes aplurality of prompt decision trees. The plurality of prompt decisiontrees includes a diagnosis prompt decision tree, a characterizationprompt decision tree, and a localization prompt decision tree. Each ofthe plurality of prompt decision trees includes a root node, a set ofinternal nodes, and a set of leaf nodes. Each root node and each of theset of internal nodes correspond to one of a plurality of prompts. Eachroot node and each of the set of internal nodes include a set ofbranches that each correspond to one of a discrete set of selectionoptions for the one of the plurality of prompts. The labelingapplication further includes application operational instructions that,when executed by at least one client device processor of the clientdevice, cause the client device to execute a labeling application.

Step 5004 includes transmitting, via the network, a medical scan to theclient device. Execution of the labeling application by the clientdevice causes the client device to, in response to receiving the medicalscan, perform the steps of FIG. 15B. Step 5006 includes receiving, viathe network, a set of labels from the client device, where the set oflabels were generated by the client device as a result of the clientdevice performing the steps of FIG. 15B in accordance with execution ofthe labeling application. Step 5008 includes populating a medical scanentry of the medical scan in a medical scan relational database based onthe set of labels.

FIG. 15B presents a flowchart illustrating a method for execution by aclient device 120 that stores labeling application data received fromthe medical scan hierarchical labeling system 3002. The labeling dataincludes the plurality of prompt decision trees and the applicationoperational instructions that, when executed by at least one processorof the client device 120, cause the client device 120 to perform thesteps below.

Step 5102 includes receiving a medical scan via a network. Step 5104includes displaying, via an interactive interface presented on a displaydevice associated with the client device for display to a userassociated with the client device, image data of the medical scan. StepS106 includes automatically determining a starting diagnosis prompt byselecting one of the set of internal nodes of the diagnosis promptdecision tree based on an anatomical region of the medical scan andfurther based on a modality of the medical scan. Step 5108 includesdisplaying, via the interactive interface, a plurality of diagnosisprompts of the diagnosis prompt decision tree, in succession, beginningwith the starting diagnosis prompt, in accordance with correspondingnodes of the diagnosis prompt decision tree until a first one of the setof leaf nodes of the diagnosis prompt decision tree is ultimatelyselected. The interactive interface progresses to each next one of theplurality of diagnosis prompts by selecting one of the set of branchesof each corresponding node in accordance with each of a plurality ofcorresponding user diagnosis selections, received via user input. Eachof the plurality of corresponding user diagnosis selections correspondsto one of the discrete set of selection options for each one of theplurality of diagnosis prompts displayed via the interactive interface.

Step 5110 includes automatically determining a starting characterizationprompt by selecting one of the set of internal nodes of thecharacterization prompt decision tree based on the anatomical region ofthe medical scan, based on the modality of the medical scan, and furtherbased on the first one of the set of leaf nodes of the diagnosis promptdecision tree. Step 5112 includes displaying, via the interactiveinterface, a plurality of characterization prompts, in succession,beginning with the starting characterization prompt, in accordance withcorresponding nodes of the characterization prompt decision tree until afirst one of the set of leaf nodes of the characterization promptdecision tree is ultimately selected. The interactive interfaceprogresses to each next one of the plurality of characterization promptsby selecting one of the set of branches of each corresponding node inaccordance with each of a plurality of corresponding usercharacterization selections, received via user input. Each of theplurality of corresponding user characterization selections correspondsto one of the discrete set of selection options for each one of theplurality of characterization prompts displayed via the interactiveinterface.

Step 5114 includes automatically determining a starting localizationprompt by selecting one of the set of internal nodes of the localizationprompt decision tree based on the anatomical region of the medical scan,and further based on the modality of the medical scan. Step S116includes displaying via the interactive interface, a plurality oflocalization prompts, in succession, beginning with the startinglocalization prompt, in accordance with corresponding nodes of thelocalization prompt decision tree until a first one of the set of leafnodes of the localization prompt decision tree is ultimately selected.The interactive interface progresses to each next one of the pluralityof localization prompts by selecting one of the set of branches of eachcorresponding node in accordance with a plurality of corresponding userlocalization selections, received via user input. Each of the pluralityof corresponding user localization selections corresponds to one of thediscrete set of selection options for each one of the plurality oflocalization prompts displayed via the interactive interface. Step 5118includes transmitting, via the network, labeling data that includes aset of labels indicating the first one of the set of leaf nodes of thediagnosis prompt decision tree, the first one of the set of leaf nodesof the characterization prompt decision tree, and the first one of theset of leaf nodes of the localization prompt decision tree.

In various embodiments, the labeling application data is transmitted bythe medical scan hierarchical labeling system 3002 to plurality ofclient devices. A plurality of medical scans are also transmitted to theplurality of client devices by the medical scan hierarchical labelingsystem 3002. A plurality of sets of leaf node identifiers for theplurality of medical scans are received by the medical scan hierarchicallabeling system 3002 from the plurality of client devices via thenetwork. A plurality of medical scan entries of the plurality of medicalscans are populated in the medical scan relational database by themedical scan hierarchical labeling system 3002. Each of the plurality ofmedical scan entries is populated based on a corresponding set of leafnode identifiers of the plurality of sets of leaf node identifiers.

In various embodiments, a computer vision model is generated by themedical scan hierarchical labeling system 3002 by performing a trainingstep on the plurality of medical scan entries. Output labels of theplurality of medical scan entries utilize structured fields of themedical scan relational database to indicate corresponding sets of leafnode identifiers. An inference function is performed by the medical scanhierarchical labeling system 3002 on a new medical scan by utilizing thecomputer vision model to generate inference data. The inference dataindicates at least one leaf node of the set of leaf nodes of thediagnosis prompt decision tree, at least one leaf node of the set ofleaf nodes of the characterization prompt decision tree, and at leastone leaf node of the set of leaf nodes of the localization promptdecision tree. The inference data is transmitted by the medical scanhierarchical labeling system 3002, via the network, to a second clientdevice for display via a second display device of the second clientdevice.

In various embodiments, execution of the labeling application by theclient device further causes the client device to display, via theinteractive interface, a differential diagnosis prompt to indicatewhether differential diagnosis is present in response to selection ofthe first one of the set of leaf nodes of the diagnosis prompt decisiontree. The client device receives a user selection to enter adifferential diagnosis in response to the differential diagnosis prompt.In response to the user selection to enter the differential diagnosis,the client device displays, via the interactive interface, a secondplurality of diagnosis prompts of the diagnosis prompt decision tree, insuccession, beginning with the starting diagnosis prompt, in accordancewith corresponding nodes of the diagnosis prompt decision tree until asecond one of the set of leaf nodes of the diagnosis prompt decisiontree is ultimately selected, where the second one of the set of leafnodes is different from the first one of the set of leaf nodes. Theclient device automatically determines a second startingcharacterization prompt by selecting one of the set of internal nodes ofthe diagnosis prompt decision tree based on the anatomical region of themedical scan, based on the modality of the medical scan, and furtherbased on the second one of the set of leaf nodes of the diagnosis promptdecision tree. The client device displays, via the interactiveinterface, a second plurality of characterization prompts, insuccession, beginning with the second starting characterization prompt,in accordance with corresponding nodes of the characterization promptdecision tree until a second one of the set of leaf nodes of thecharacterization prompt decision tree is ultimately selected. The clientdevice displays via the interactive interface, a second plurality oflocalization prompts, in succession, beginning with the startinglocalization prompt, in accordance with corresponding nodes of thelocalization prompt decision tree until a second one of the set of leafnodes of the localization prompt decision tree is ultimately selected.The set of labels further indicates the second one of the set of leafnodes of the diagnosis prompt decision tree, the second one of the setof leaf nodes of the characterization prompt decision tree, and thesecond one of the set of leaf nodes of the localization prompt decisiontree.

In various embodiments, execution of the labeling application by theclient device further causes the client device to display, via theinteractive interface, the differential diagnosis prompt to indicatewhether differential diagnosis is necessary in response to selection ofeach one of a selected plurality of leaf nodes of the set of leaf nodesof the diagnosis prompt decision tree, until receiving a user selectionindicating further differential diagnosis is not present. The clientdevice automatically determines a plurality of starting characterizationprompts corresponding to the selected plurality of leaf nodes of the setof leaf nodes of the diagnosis prompt decision tree. The client devicedisplays, via the interactive interface, a plurality characterizationprompts for each of the selected plurality of leaf nodes of the set ofleaf nodes of the diagnosis prompt decision tree, in succession,beginning with each corresponding one of the plurality of startingcharacterization prompts, in accordance with corresponding nodes of thecharacterization prompt decision tree until each of a correspondingselected plurality of the set of leaf nodes of the characterizationprompt decision tree are ultimately selected. The client devicedisplays, via the interactive interface, a plurality of localizationprompts for each of the selected plurality of leaf nodes of the set ofleaf nodes of the diagnosis prompt decision tree, in succession,beginning with the starting localization prompt, in accordance withcorresponding nodes of the characterization prompt decision tree untileach of a corresponding selected plurality of the set of leaf nodes ofthe localization prompt decision tree are ultimately selected. The setof labels further indicates the selected plurality of the set of leafnodes of the diagnosis prompt decision tree, the corresponding selectedplurality of the set of leaf nodes of the characterization promptdecision tree, and the corresponding selected plurality of the set ofleaf nodes of the localization prompt decision tree.

In various embodiments, execution of the labeling application by theclient device further causes the client device to display, via theinteractive interface, an abnormality present prompt to indicate whetheran abnormality is present in response to receiving the medical scan. Auser selection indicating that no abnormality is present is received bythe client device in response to the abnormality present prompt. Theclient device determines to forego display of the plurality of diagnosisprompts, the plurality of characterization prompts, and the plurality oflocalization prompts in response to the user selection indicating thatno abnormality is present. The set of labels indicates that the medicalscan is normal in response to the user selection indicating that noabnormality is present.

In various embodiments, one of the plurality of characterization promptsincludes a prompt to the user to indicate whether at least one secondaryfinding is present. In response to the client device receiving a userselection indicating at least one secondary finding is present,successive ones of the plurality of characterization prompts presentedby the client device correspond to characterization of the at least onesecondary finding. The labeling data indicates at least one leaf nodeindicating the at least one secondary finding of a primary diagnosisindicated by the first one of the set of leaf nodes of the diagnosisprompt decision tree.

In various embodiments, execution of the labeling application by theclient device further causes the client device to, in response toreceiving a user selection indicating at least one secondary findings ispresent, automatically determine a second starting diagnosis prompt byselecting one of the set of internal nodes of the diagnosis promptdecision tree based on the first one of the set of leaf nodes of thediagnosis prompt decision tree. The client device displays, via theinteractive interface, a plurality of diagnosis prompts of the diagnosisprompt decision tree, in succession, beginning with the startingdiagnosis prompt, in accordance with corresponding nodes of thediagnosis prompt decision tree until a second one of the set of leafnodes of the diagnosis prompt decision tree is ultimately selected,where the second one of the set of leaf nodes of the diagnosis promptdecision tree is different from the first one of the set of leaf nodesof the diagnosis prompt decision tree. The client device automaticallydetermines a second starting characterization prompt by selecting one ofthe set of internal nodes of the diagnosis prompt decision tree based onthe anatomical region of the medical scan, based on the modality of themedical scan, and further based on the second one of the set of leafnodes of the diagnosis prompt decision tree. The client device displays,via the interactive interface, a second plurality of characterizationprompts, in succession, beginning with the second startingcharacterization prompt, in accordance with corresponding nodes of thecharacterization prompt decision tree until a second one of the set ofleaf nodes of the characterization prompt decision tree is ultimatelyselected. The client device displays, via the interactive interface, asecond plurality of localization prompts, in succession, beginning withthe starting localization prompt, in accordance with corresponding nodesof the localization prompt decision tree until a second one of the setof leaf nodes of the localization prompt decision tree is ultimatelyselected. The set of labels further indicates the second one of the setof leaf nodes of the diagnosis prompt decision tree, the second one ofthe set of leaf nodes of the characterization prompt decision tree, andthe second one of the set of leaf nodes of the localization promptdecision tree. The labeling data indicates the first one of the set ofleaf nodes of the diagnosis prompt decision tree corresponds to aprimary finding. The labeling data further indicates the second one ofthe set of leaf nodes of the diagnosis prompt decision tree, the secondone of the set of leaf nodes of the characterization prompt decisiontree, and the second one of the set of leaf nodes of the localizationprompt decision tree correspond to a secondary finding of the primaryfinding. In various embodiments, the primary diagnosis indicated by thefirst one of the set of leaf nodes of the diagnosis prompt decision treecorresponds to a brain tumor, and the at least one secondary findingindicated by the at least one leaf node correspond to a fracture and/ora brain bleed.

In various embodiments, a subset of the set of internal nodes of thelocalization prompt decision tree has a plurality of mandatory branches.One of the plurality of localization prompts displayed via theinteractive interface corresponds to one of the set of internal nodesincluded in the subset. Execution of the labeling application by theclient device further causes the client device to display ones of theplurality of localization prompts corresponding to nodes branching fromeach of the plurality of mandatory branches of the one of the set ofinternal nodes included in the subset until a plurality of leaf nodes ofthe set of leaf nodes of the localization prompt decision tree are eachreached in succession. The set of labels includes the plurality of leafnodes of the set of leaf nodes of the localization prompt decision tree.In various embodiments, the medical scan is a head CT scan, and theplurality of leaf nodes of the set of leaf nodes indicates at least onelobe and at least one compartment. In various embodiments, the pluralityof leaf nodes of the set of leaf nodes indicates exactly one lobe andexactly one compartment.

In various embodiments, automatically determining the startingcharacterization prompt is further based on the first one of the set ofleaf nodes of the diagnosis prompt decision tree. Automaticallydetermining the starting localization prompt is further based on thefirst one of the set of leaf nodes of the diagnosis prompt decisiontree.

In various embodiments, the medical scan includes a plurality of imageslices. Execution of the labeling application by the client devicefurther causes the client device to display, via the interactiveinterface, a slice selection prompt. A user selection that indicates aselected subset of the plurality of images slices of the medical scan isreceived by the client device. The starting localization prompt isfurther selected based on the selected subset of the plurality of imageslices. In various embodiments, automatically determining the startingdiagnosis prompt is further based on the selected subset of theplurality of image slices.

In various embodiments, execution of the labeling application by theclient device further causes the client device to display, via theinteractive interface, an urgency prompt. A user selection thatindicates a urgency ranking of the medical scan is received by theclient device in response to the urgency prompt. The set of labelsfurther indicates the urgency ranking. In various embodiments, thelabeling application data further includes a plurality of additionalquestion sets. Execution of the labeling application by the clientdevice further causes the client device to automatically determine anadditional question set from the plurality of additional question setsbased on based on the anatomical region of the medical scan and furtherbased on the modality of the medical scan.

In various embodiments, execution of the labeling application by theclient device causes the client device to, in response to receiving themedical scan generate a plurality of partitioned portions of the medicalscan in accordance with a corresponding plurality of anatomical regionsby performing a pre-processing step. The client device determines astarting diagnosis prompt, a starting characterization prompt, and astarting localization prompt for each of the plurality of partitionedportions based on one of the corresponding plurality of anatomicalregions. Each one of the plurality of partitioned portions of themedical scan are displayed by the client device in succession. For eachdisplayed one of the plurality of portioned portions of the medicalscan, a plurality of diagnosis prompts are displayed by the clientdevice in conjunction with the displayed one of the plurality ofpartitioned portions of the medical scan, in succession, beginning witha corresponding one of the plurality of starting diagnosis prompts,until one of the set of leaf nodes of the diagnosis prompt decision treeis ultimately selected for the displayed one of the plurality ofpartitioned portions of the medical scan. For each displayed one of theplurality of portioned portions of the medical scan, the client devicedisplays a plurality of characterization prompts in conjunction with thedisplayed one of the plurality of partitioned portions of the medicalscan, in succession, beginning with a corresponding one of the pluralityof starting characterization prompts, until one of the set of leaf nodesof the characterization prompt decision tree is ultimately selected forthe displayed one of the plurality of partitioned portions of themedical scan. For each displayed one of the plurality of portionedportions of the medical scan, the client device displays a plurality oflocalization prompts in conjunction with the displayed one of theplurality of partitioned portions of the medical scan, in succession,beginning with a corresponding one of the plurality of startinglocalization prompts, until one of the set of leaf nodes of thelocalization prompt decision tree is ultimately selected for thedisplayed one of the plurality of partitioned portions of the medicalscan. The set of labels indicates the one of the set of leaf nodes ofthe diagnosis prompt decision tree for each displayed one of theplurality of portioned portions of the medical scan, the first one ofthe set of leaf nodes of the characterization prompt decision tree foreach displayed one of the plurality of portioned portions of the medicalscan, and the first one of the set of leaf nodes of the localizationprompt decision tree for each displayed one of the plurality ofportioned portions of the medical scan.

Some or all of the steps performed by the client device 120 can instead,or additionally, be performed by the medical scan hierarchical labelingsystem 3002, for example, in response to transmissions from the clientdevice that indicate user input to the interactive interface. Results ofsome or all of these steps are transmitted back to the client device,and the interactive interface can display some or all of the promptsand/or other information to the user based on these results receivedfrom the medical scan hierarchical labeling system 3002.

FIG. 16 presents a flowchart illustrating a method for execution by amedical scan labeling quality assurance system 3004 that storesexecutional instructions that, when executed by at least one processor,cause the medical scan labeling quality assurance system 3004 to performthe steps below.

Step 5202 includes selecting a set of users from a user database inresponse to determining a scheduled interval has elapsed. The userdatabase includes a plurality of user profiles corresponding to aplurality of users, and the set of users is selected from the pluralityof users. Step 5204 includes selecting a set of medical scans from amedical scan database. The medical scan database includes a plurality ofmedical scans, and the set of medical scans is selected from theplurality of medical scans. Step 5206 includes transmitting the set ofmedical scans, via a network, to a set of client devices associated withthe set of users. The set of medical scans are displayed to the set ofusers via a first interactive interface displayed by a set of displaydevices corresponding to the set of client devices. Step 5208 includesreceiving a set of labeling data from each of the set of client devicesvia the network. Each set of labeling data is generated by acorresponding one of the set of client devices, and each set of labelingdata includes labeling data for each of the set of medical scans. Thelabeling data for each of the set of medical scans is generated by thecorresponding one of the set of client devices in response to at leastone prompt to provide the labeling data via the first interactiveinterface in conjunction with display of the each of the set of medicalsscans.

Step 5210 includes selecting an expert user from the user database,where the expert user is not included in the set of users. Step 5212includes transmitting the set of medical scans, via the network, to anexpert client device associated with the expert user. The set of medicalscans are displayed to the expert user via a second interactiveinterface displayed by an expert display device corresponding to theexpert user. Step 5214 includes receiving a golden set of labeling datafrom the expert client device via the network. The golden set oflabeling data is generated by the expert client device and includesgolden labeling data for each of the set of medical scans. The goldenlabeling data for each of the set of medical scans is generated by theexpert client device in response to at least one prompt to provide thegolden labeling data via the second interactive interface in conjunctionwith display of the each of the set of medicals scans.

Step 5216 includes generating a set of performance score data bygenerating performance score data for each corresponding set of labelingdata by comparing the labeling data of each set of labeling data to thegolden labeling data of the set of golden labeling data. Step 5218includes assigning each performance score data of the set of performancescore data to a corresponding one of the set of users that generated thecorresponding set of labeling data. Step S220 includes updating each ofa set of user profile entries in the user database for eachcorresponding one of the set of users based on the performance scoredata of the set of performance score data assigned to the correspondingone of the set of users. Step 5222 includes transmitting eachperformance score data of the set of performance score data to acorresponding one of the set of client devices for display, via thefirst interactive interface, to a corresponding one of the set of usersto which the each performance score data is assigned.

In various embodiments, the medical scan labeling quality assurancesystem 3004 transmits each set of labeling data to the expert clientdevice. The set of labeling data is displayed to the displayed to theexpert user via the second interactive interface in conjunction withdisplay of the set of medical scans. In various embodiments, medicalscan labeling quality assurance system 3004 receives a set of correctiondata from each of the set of client devices. Each set of correction datacorresponds to one set of labeling data, and each correction data ofeach set of correction data corresponds to one of the set of medicalscans. Each set of correction data is generated by the expert clientdevice in response to at least one additional prompt to provide the eachof the set of correction data via the second interactive interface inconjunction with display of each corresponding set of labeling data. Theset of performance score data is further generated based on the set ofcorrection data. The medical scan labeling quality assurance system 3004transmits each correction data of the set of correction data to acorresponding one of the set of client devices for display, via thefirst interactive interface, in conjunction with set of medical scansand the each performance score data.

In various embodiments, each set of correction data includes commentdata, and the comment data is generated by the expert client device inresponse to text entered by the expert user in at least one text boxdisplayed by the second interactive interface. In various embodiments,medical scan labeling quality assurance system 3004 transmits eachcorrection data of the set of correction data to all of the set ofclient devices for display in conjunction with the set of medical scans.In various embodiments, the medical scan labeling quality assurancesystem 3004 selects a second set of users from the user database. A setdifference between the second set of users and the set of users and isnon-null, where at least one of the second set of users is not includedin the first set of users. Alternatively or in addition, an intersectionbetween the set of users and the second set of users is non-null. Theset of medical scans and the set of correction data is sent to each oneof a second set of client devices corresponding to the second set ofusers for display, via a third interactive interface, in conjunctionwith set of medical scans.

In various embodiments, the medical scan labeling quality assurancesystem 3004 determines a common error in the sets of labeling data forat least one of the set of medical scans based on the set of goldenlabeling data. The medical scan labeling quality assurance system 3004identifies at least one set of labeling data that includes the commonerror. Only the at least one of the set of labeling data that includesthe common error and only the at least one of the of the set of medicalscans corresponding to the common error are transmitted to the set ofmedical scans, where the third interactive interface indicates thecommon error. In various embodiments, the medical scan labeling qualityassurance system 3004 generates labeling commonality data by comparingthe sets of labeling data to each other to determine similar ones of thesets of labeling data for at least one of the set of medical scans. Themedical scan labeling quality assurance system 3004 generates commonerror data based on comparing the labeling commonality data to thegolden labeling data to determine a proper subset of the similar ones ofthe sets of labeling data that compares unfavorably to the goldenlabeling data. The common error is determined based on the common errordata. In various embodiments, the common error is identified by theexpert client device in response to at least one additional prompt toprovide the common error data via the second interactive interface.

In various embodiments the medical scan labeling quality assurancesystem 3004 determines that the scheduled interval has elapsed sincetransmission of the set of medical scans to the set of client devices.The medical scan labeling quality assurance system 3004 selects a secondset of users from the user database after the scheduled interval haselapsed since selection of the set of users in response to determiningthe scheduled time interval has elapsed. The medical scan labelingquality assurance system 3004 selects a second set of medical scans fromthe medical scan database in response to determining the scheduled timeinterval has elapsed. An intersection between the set of medical scansand the second set of medical scans is null. The medical scan labelingquality assurance system 3004 transmits the second set of medical scans,via a network, to a second set of client devices associated with thesecond set of users. The second set of medical scans are displayed tothe second set of users via the first interactive interface displayed bya second set of display devices corresponding to the second set ofclient devices.

A second set of labeling data is received from each of the second set ofclient devices via the network. Each second set of labeling data isgenerated by a corresponding one of the second set of client devices,and each second set of labeling data includes second labeling data foreach of the second set of medical scans. The second labeling data foreach of the second set of medical scans is generated by thecorresponding one of the second set of client devices in response to atleast one prompt to provide the second labeling data via the firstinteractive interface in conjunction with display of the each of thesecond set of medicals scans. The medical scan labeling qualityassurance system 3004 selects a second expert user from the userdatabase, where the second expert user is not included in the second setof users. The medical scan labeling quality assurance system 3004transmits the second set of medical scans, via the network, to a secondexpert client device associated with the second expert user. The secondset of medical scans are displayed to the second expert user via asecond interactive interface displayed by a second expert display devicecorresponding to the second expert user.

The medical scan labeling quality assurance system 3004 receives asecond golden set of labeling data from the second expert client devicevia the network. The second golden set of labeling data is generated bythe second expert client device and includes second golden labeling datafor each of the second set of medical scans. The second golden labelingdata for each of the second set of medical scans is generated by thesecond expert client device in response to at least one prompt toprovide the second golden labeling data via the second interactiveinterface in conjunction with display of the each of the second set ofmedicals scans.

The medical scan labeling quality assurance system 3004 generates asecond set of performance score data by generating second performancescore data for each corresponding second set of labeling data bycomparing the second labeling data of each second set of labeling datato the second golden labeling data of the second set of global labelingdata. The medical scan labeling quality assurance system 3004 assignseach second performance score data of the second set of performancescore data to a corresponding one of the second set of users thatgenerated the corresponding second set of labeling data. The medicalscan labeling quality assurance system 3004 updates each of a second setof user profile entries in the user database for each corresponding oneof the second set of users based on the second performance score data ofthe second set of performance score data assigned to the correspondingone of the second set of users. The medical scan labeling qualityassurance system 3004 transmits each second performance score data ofthe second set of performance score data to a corresponding one of thesecond set of client devices for display, via the first interactiveinterface, to a corresponding one of the second set of users to whichthe each second performance score data is assigned.

In various embodiments, the medical scan labeling quality assurancesystem 3004 identifies a subset of the set of users with performancescore data that compares unfavorably to a performance score data upperbound. Selecting the second set of users includes selecting only ones ofthe set of users identified in the subset. In various embodiments, themedical scan labeling quality assurance system 3004 identifies a secondsubset of the plurality of users with performance score data thatcompares unfavorably to a performance score data upper bound. The secondsubset includes at least one user of the plurality of users that is notincluded in the set of users. Selecting the second set of users includesselecting only ones of the set of users identified in the second subset.

In various embodiments, the medical scan labeling quality assurancesystem 3004 identifies a second subset of the set of users withperformance score data that compares favorably to the performance scoredata upper bound. The medical scan labeling quality assurance system3004 updates user profile entries in the user database for each one ofthe second subset of the set of users to indicate an expert status foreach one of the second subset of the set of users with in response tothe performance score data that compares favorably to the performancescore data upper bound. Selecting the expert user and the second expertuser includes determining that the expert user and the second expertuser have corresponding user profile entries indicating the expertstatus. In various embodiments, identifying the second subset of the setof users further includes determining for each of the set of users, anumber of medical scans labeled with performance score data that comparefavorably to the performance score data upper bound. Only ones of theset of users with a corresponding number that compares favorably to athreshold are included in the second set of users.

In various embodiments, the medical scan labeling quality assurancesystem 3004 identifies a subset of the set of users with performancescore data that compare favorably to a performance score data lowerbound. Selecting the second set of users includes selecting only ones ofthe set of users identified in the subset. The medical scan labelingquality assurance system 3004 identifies a second subset of the set ofusers with performance score data that compare unfavorably to aperformance score data lower bound. The medical scan labeling qualityassurance system 3004 transmits a notification to a subset of the set ofclient devices corresponding to each of the second subset of the set ofusers indicating a dismissal from labeling of future medical scans.

In various embodiments, the medical scan labeling quality assurancesystem 3004 selects the set of medical scans randomly. In variousembodiments, the medical scan labeling quality assurance system 3004selects the set of medical scans based on a set of determined selectionrequirements, where the set of determined selection requirementsindicates at least one modality and further indicates at least oneanatomical region. In various embodiments, the medical scan labelingquality assurance system 3004 selects the set of medical scans based onan urgency rating of the set of medical scans.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may further be used herein, the term “associatedwith”, includes direct and/or indirect coupling of separate items and/orone item being embedded within another item. As may still further beused herein, the term “automatically” refers to an action causeddirectly by a processor of a computer network in response to atriggering event and particularly without human interaction.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing device” and/or “processing unit” maybe a single processing device or a plurality of processing devices. Sucha processing device may be a microprocessor, micro-controller, digitalsignal processor, graphics processing unit, microcomputer, centralprocessing unit, field programmable gate array, programmable logicdevice, state machine, logic circuitry, analog circuitry, digitalcircuitry, and/or any device that manipulates signals (analog and/ordigital) based on hard coding of the circuitry and/or operationalinstructions. The processing module, module, processing circuit, and/orprocessing unit may be, or further include, memory and/or an integratedmemory element, which may be a single memory device, a plurality ofmemory devices, and/or embedded circuitry of another processing module,module, processing circuit, and/or processing unit. Such a memory devicemay be a read-only memory, random access memory, volatile memory,non-volatile memory, static memory, dynamic memory, flash memory, cachememory, and/or any device that stores digital information. Note that ifthe processing module, module, processing circuit, and/or processingunit includes more than one processing device, the processing devicesmay be centrally located (e.g., directly coupled together via a wiredand/or wireless bus structure) or may be distributedly located (e.g.,cloud computing via indirect coupling via a local area network and/or awide area network). Further note that if the processing module, module,processing circuit, and/or processing unit implements one or more of itsfunctions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, and/or processing unit executes, hard coded and/oroperational instructions corresponding to at least some of the stepsand/or functions illustrated in one or more of the Figures and/ordescribed herein. Such a memory device or memory element can be includedin an article of manufacture. While the processing module, module,processing circuit, and/or processing unit device may be a generalpurpose computing device, the execution of the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit configures such a general purposecomputing device as a special purpose computing device to implement thecorresponding steps and/or functions illustrated in one or more of theFigures and/or described herein. In particular, the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit implement acts and algorithms performedby the processing module, module, processing circuit, and/or processingunit. Such acts and algorithms can be identified by name, can beillustrated via flowchart and/or described in words.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

The term “system” is used in the description of one or more of theembodiments. A system implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A system may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a system may containone or more sub-system, each of which may be one or more systems.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form a solidstate memory, a hard drive memory, cloud memory, thumb drive, servermemory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A medical scan labeling quality assurance system,comprising: a medical scan database that includes a plurality of medicalscans; a user database that includes a plurality of user profilescorresponding to a plurality of users of the medical scan labelingquality assurance system; a processing system that includes a processor;and a memory that stores executable instructions that, when executed bythe processing system, cause the medical scan labeling quality assurancesystem to: select a set of users from the user database in response todetermining a scheduled interval has elapsed; select a set of medicalscans from the medical scan database; transmit the set of medical scans,via a network, to a set of client devices associated with the set ofusers, wherein the set of medical scans are displayed to the set ofusers via a first interactive interface displayed by a set of displaydevices corresponding to the set of client devices; receive a set oflabeling data from each of the set of client devices via the network,wherein each set of labeling data is generated by a corresponding one ofthe set of client devices, wherein each set of labeling data includeslabeling data for each of the set of medical scans, and wherein thelabeling data for each of the set of medical scans is generated by thecorresponding one of the set of client devices in response to at leastone prompt to provide the labeling data via the first interactiveinterface in conjunction with display of the each of the set of medicalsscans; select an expert user from the user database, wherein the expertuser is not included in the set of users; transmit the set of medicalscans, via the network, to an expert client device associated with theexpert user, wherein the set of medical scans are displayed to theexpert user via a second interactive interface displayed by an expertdisplay device corresponding to the expert user; receive a golden set oflabeling data from the expert client device via the network, wherein thegolden set of labeling data is generated by the expert client device andincludes golden labeling data for each of the set of medical scans, andwherein the golden labeling data for each of the set of medical scans isgenerated by the expert client device in response to at least one promptto provide the golden labeling data via the second interactive interfacein conjunction with display of the each of the set of medicals scans;generate a set of performance score data by generating performance scoredata for each corresponding set of labeling data by comparing thelabeling data of each set of labeling data to the golden labeling dataof the set of golden labeling data; assign each performance score dataof the set of performance score data to a corresponding one of the setof users that generated the corresponding set of labeling data; updateeach of a set of user profile entries in the user database for eachcorresponding one of the set of users based on the performance scoredata of the set of performance score data assigned to the correspondingone of the set of users; and transmit each performance score data of theset of performance score data to a corresponding one of the set ofclient devices for display, via the first interactive interface, to acorresponding one of the set of users to which the each performancescore data is assigned.
 2. The medical scan labeling quality assurancesystem of claim 1, wherein the executable instructions, when executed bythe processing system, further cause the medical scan labeling qualityassurance system to transmit each set of labeling data to the expertclient device, wherein set of labeling data is displayed to thedisplayed to the expert user via the second interactive interface inconjunction with display of the set of medical scans.
 3. The medicalscan labeling quality assurance system of claim 2, wherein theexecutable instructions, when executed by the processing system, furthercause the medical scan labeling quality assurance system to: receive aset of correction data from each of the set of client devices, whereineach set of correction data corresponds to one set of labeling data,wherein each correction data of each set of correction data correspondsto one of the set of medical scans, wherein each set of correction datais generated by the expert client device in response to at least oneadditional prompt to provide the each of the set of correction data viathe second interactive interface in conjunction with display of eachcorresponding set of labeling data, and wherein the set of performancescore data is further generated based on the set of correction data; andtransmit each correction data of the set of correction data to acorresponding one of the set of client devices for display, via thefirst interactive interface, in conjunction with set of medical scansand the each performance score data.
 4. The medical scan labelingquality assurance system of claim 3, wherein each set of correction dataincludes comment data, and wherein the comment data is generated by theexpert client device in response to text entered by the expert user inat least one text box displayed by the second interactive interface. 5.The medical scan labeling quality assurance system of claim 3, whereinthe executable instructions, when executed by the processing system,further cause the medical scan labeling quality assurance system to:transmit each correction data of the set of correction data to all ofthe set of client devices for display in conjunction with the set ofmedical scans.
 6. The medical scan labeling quality assurance system ofclaim 3, wherein the executable instructions, when executed by theprocessing system, further cause the medical scan labeling qualityassurance system to: select a second set of users from the userdatabase, wherein a set difference between the second set of users andthe set of users is non-null; and transmit the set of medical scans andthe set of correction data to each one of a second set of client devicescorresponding to the second set of users for display, via a thirdinteractive interface, in conjunction with set of medical scans.
 7. Themedical scan labeling quality assurance system of claim 6, wherein theexecutable instructions, when executed by the processing system, furthercause the medical scan labeling quality assurance system to: determine acommon error in the sets of labeling data for at least one of the set ofmedical scans based on the set of golden labeling data; and identify atleast one set of labeling data that includes the common error; whereinonly the at least one of the set of labeling data that includes thecommon error and only the at least one of the of the set of medicalscans corresponding to the common error are transmitted to the secondset of medical scans, and wherein the third interactive interfaceindicates the common error.
 8. The medical scan labeling qualityassurance system of claim 7, wherein the executable instructions, whenexecuted by the processing system, further cause the medical scanlabeling quality assurance system to: generate labeling commonality databy comparing the sets of labeling data to each other to determinesimilar ones of the sets of labeling data for at least one of the set ofmedical scans; and generate common error data based on comparing thelabeling commonality data to the golden labeling data to determine aproper subset of the similar ones of the sets of labeling data thatcompares unfavorably to the golden labeling data; wherein the commonerror is determined based on the common error data.
 9. The medical scanlabeling quality assurance system of claim 7, wherein the common erroris identified by the expert client device in response to at least oneadditional prompt to provide the common error data via the secondinteractive interface.
 10. The medical scan labeling quality assurancesystem of claim 1, wherein the executable instructions, when executed bythe processing system, further cause the medical scan labeling qualityassurance system to: determine that the scheduled interval has elapsedsince transmission of the set of medical scans to the set of clientdevices; select a second set of users from the user database after thescheduled interval has elapsed since selection of the set of users inresponse to determining the scheduled time interval has elapsed; selecta second set of medical scans from the medical scan database in responseto determining the scheduled time interval has elapsed, wherein anintersection between the set of medical scans and the second set ofmedical scans is null; transmit the second set of medical scans, via anetwork, to a second set of client devices associated with the secondset of users, wherein the second set of medical scans are displayed tothe second set of users via the first interactive interface displayed bya second set of display devices corresponding to the second set ofclient devices; receive a second set of labeling data from each of thesecond set of client devices via the network, wherein each second set oflabeling data is generated by a corresponding one of the second set ofclient devices, wherein each second set of labeling data includes secondlabeling data for each of the second set of medical scans, and whereinthe second labeling data for each of the second set of medical scans isgenerated by the corresponding one of the second set of client devicesin response to at least one prompt to provide the second labeling datavia the first interactive interface in conjunction with display of theeach of the second set of medicals scans; select a second expert userfrom the user database, wherein the second expert user is not includedin the second set of users; transmit the second set of medical scans,via the network, to a second expert client device associated with thesecond expert user, wherein the second set of medical scans aredisplayed to the second expert user via a second interactive interfacedisplayed by a second expert display device corresponding to the secondexpert user; receive a second golden set of labeling data from thesecond expert client device via the network, wherein the second goldenset of labeling data is generated by the second expert client device andincludes second golden labeling data for each of the second set ofmedical scans, and wherein the second golden labeling data for each ofthe second set of medical scans is generated by the second expert clientdevice in response to at least one prompt to provide the second goldenlabeling data via the second interactive interface in conjunction withdisplay of the each of the second set of medicals scans; generate asecond set of performance score data by generating second performancescore data for each corresponding second set of labeling data bycomparing the second labeling data of each second set of labeling datato the second golden labeling data of the second set of global labelingdata; assign each second performance score data of the second set ofperformance score data to a corresponding one of the second set of usersthat generated the corresponding second set of labeling data; updateeach of a second set of user profile entries in the user database foreach corresponding one of the second set of users based on the secondperformance score data of the second set of performance score dataassigned to the corresponding one of the second set of users; andtransmit each second performance score data of the second set ofperformance score data to a corresponding one of the second set ofclient devices for display, via the first interactive interface, to acorresponding one of the second set of users to which the each secondperformance score data is assigned.
 11. The medical scan labelingquality assurance system of claim 10, wherein the executableinstructions, when executed by the processing system, further cause themedical scan labeling quality assurance system to: identify a subset ofthe set of users with performance score data that compares unfavorablyto a performance score data upper bound; wherein selecting the secondset of users includes selecting only ones of the set of users identifiedin the subset.
 12. The medical scan labeling quality assurance system ofclaim 10, wherein the executable instructions, when executed by theprocessing system, further cause the medical scan labeling qualityassurance system to: identify a second subset of the plurality of userswith performance score data that compares unfavorably to a performancescore data upper bound, wherein the second subset includes at least oneuser of the plurality of users that is not included in the set of users;wherein selecting the second set of users includes selecting only onesof the set of users identified in the second subset.
 13. The medicalscan labeling quality assurance system of claim 10, wherein theexecutable instructions, when executed by the processing system, furthercause the medical scan labeling quality assurance system to: identify asecond subset of the set of users with performance score data thatcompares favorably to the performance score data upper bound; and updateuser profile entries in the user database for each one of the secondsubset of the set of users to indicate an expert status for each one ofthe second subset of the set of users in response to the performancescore data that compares favorably to the performance score data upperbound; wherein selecting the expert user and the second expert userincludes determining that the expert user and the second expert userhave corresponding user profile entries indicating the expert status.14. The medical scan labeling quality assurance system of claim 13,wherein identifying the second subset of the set of users furtherincludes: determining for each of the set of users, a number of medicalscans labeled with performance score data that compare favorably to theperformance score data upper bound; wherein only ones of the set ofusers with a corresponding number that compares favorably to a thresholdare included in the second set of users.
 15. The medical scan labelingquality assurance system of claim 10, wherein the executableinstructions, when executed by the processing system, further cause themedical scan labeling quality assurance system to: identify a subset ofthe set of users with performance score data that compare favorably to aperformance score data lower bound, wherein selecting the second set ofusers includes selecting only ones of the set of users identified in thesubset; identify a second subset of the set of users with performancescore data that compare unfavorably to a performance score data lowerbound; and transmit a notification to a subset of the set of clientdevices corresponding to each of the second subset of the set of usersindicating a dismissal from labeling of future medical scans.
 16. Themedical scan labeling quality assurance system of claim 1, wherein theexecutable instructions, when executed by the processing system, furthercause the medical scan labeling quality assurance system to select theset of medical scans randomly.
 17. The medical scan labeling qualityassurance system of claim 1, wherein the executable instructions, whenexecuted by the processing system, further cause the medical scanlabeling quality assurance system to select the set of medical scansbased on a set of determined selection requirements, wherein the set ofdetermined selection requirements indicates at least one modality andfurther indicates at least one anatomical region.
 18. The medical scanlabeling quality assurance system of claim 1, wherein the executableinstructions, when executed by the processing system, further cause themedical scan labeling quality assurance system to select the set ofmedical scans based on an urgency rating of the set of medical scans.19. A method for execution by a medical scan labeling quality assurancesystem, comprising: selecting a set of users from a user database inresponse to determining a scheduled interval has elapsed; selecting aset of medical scans from a medical scan database; transmitting the setof medical scans, via a network, to a set of client devices associatedwith the set of users, wherein the set of medical scans are displayed tothe set of users via a first interactive interface displayed by a set ofdisplay devices corresponding to the set of client devices; receiving aset of labeling data from each of the set of client devices via thenetwork, wherein each set of labeling data is generated by acorresponding one of the set of client devices, wherein each set oflabeling data includes labeling data for each of the set of medicalscans, and wherein the labeling data for each of the set of medicalscans is generated by the corresponding one of the set of client devicesin response to at least one prompt to provide the labeling data via thefirst interactive interface in conjunction with display of the each ofthe set of medicals scans; selecting an expert user from the userdatabase, wherein the expert user is not included in the set of users;transmitting the set of medical scans, via the network, to an expertclient device associated with the expert user, wherein the set ofmedical scans are displayed to the expert user via a second interactiveinterface displayed by an expert display device corresponding to theexpert user; receiving a golden set of labeling data from the expertclient device via the network, wherein the golden set of labeling datais generated by the expert client device and includes golden labelingdata for each of the set of medical scans, and wherein the goldenlabeling data for each of the set of medical scans is generated by theexpert client device in response to at least one prompt to provide thegolden labeling data via the second interactive interface in conjunctionwith display of the each of the set of medicals scans; generating a setof performance score data by generating performance score data for eachcorresponding set of labeling data by comparing the labeling data ofeach set of labeling data to the golden labeling data of the set ofgolden labeling data; assigning each performance score data of the setof performance score data to a corresponding one of the set of usersthat generated the corresponding set of labeling data; updating each ofa set of user profile entries in the user database for eachcorresponding one of the set of users based on the performance scoredata of the set of performance score data assigned to the correspondingone of the set of users; and transmitting each performance score data ofthe set of performance score data to a corresponding one of the set ofclient devices for display, via the first interactive interface, to acorresponding one of the set of users to which the each performancescore data is assigned.
 20. The method of claim 19, further comprisingtransmitting each set of labeling data to the expert client device,wherein set of labeling data is displayed to the displayed to the expertuser via the second interactive interface in conjunction with display ofthe set of medical scans.